> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vast.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade the NVIDIA Driver

> Bring a host machine up to the minimum NVIDIA driver version Vast asks for, built by DKMS and held at that version.

> [!CAUTION]
>
> Before you start: [schedule a maintenance window](https://docs.vast.ai/host/set-maintenance-window.md) at least **48 hours** in advance.
>
> You are responsible for the maintenance and any resulting interruption. Work performed outside the scheduled maintenance window may be treated as an **operational failure**.
>
> If you are unsure whether this operation could cause data loss, stop here.

Vast asks hosts to run at least a minimum NVIDIA driver version, set by GPU
generation and, on Turing and newer, by the driver the machine runs now. When
Vast raises a minimum, this table is updated; follow this page again to move to
it. The steps are the same every time.

| GPUs | Compute capability | Current driver | Minimum driver version |
| - | - | - | - |
| Turing and newer | 7.5 and higher | 595.xx | 595.91.07 |
| Turing and newer | 7.5 and higher | Any other | 610.57.04 |
| Maxwell, Pascal and Volta | below 7.5 | Any | 580.178.04 |

A machine at or above its minimum needs nothing from this page. For a machine
below it, this page installs the minimum version from NVIDIA's apt repository,
has DKMS build its kernel module, and holds it so nothing upgrades it. With
DKMS, a kernel update rebuilds the same driver for the new kernel instead of
changing the driver.

> [!NOTE]
>
> Driver branches newer than the ones in the table, such as 620, are not directly
> supported yet, though a machine on one may still run on the platform. Hosts on
> an unlisted branch are expected to keep the driver on the newest release of that
> branch, for its latest security fixes. This page does not change it.

> [!WARNING]
>
> Before step 3, [unlist the machine](https://docs.vast.ai/host/cli/unlist-machine.md) and make sure it
> has no instances; a maintenance window does neither. From step 3 until the
> reboot in step 6 the GPUs are unusable, and the reboot stops every instance.

## 1. Check the current driver

```bash theme={null}
nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv,noheader
```

```text nocopy theme={null}
NVIDIA GeForce RTX 4090, 8.9, 580.178.04
NVIDIA GeForce RTX 4090, 8.9, 580.178.04
```

The second column is the compute capability and the third is the current driver.
Set the minimum from the table row that matches both; it is also the version
this page installs. The later commands read these two variables, so if you log
out, set them again before you continue.

Compute capability 7.5 or higher, current driver 595.xx:

```bash theme={null}
DRIVER_VERSION=595.91.07; DRIVER_PACKAGE=nvidia-open
```

Compute capability 7.5 or higher, any other driver:

```bash theme={null}
DRIVER_VERSION=610.57.04; DRIVER_PACKAGE=nvidia-open
```

Below 7.5:

```bash theme={null}
DRIVER_VERSION=580.178.04; DRIVER_PACKAGE=nvidia-driver-580
```

Check whether the machine already meets the minimum:

```bash theme={null}
dpkg --compare-versions "$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | sort -u)" ge "$DRIVER_VERSION" && echo "At or above $DRIVER_VERSION: nothing to do" || echo "Below $DRIVER_VERSION: continue to step 2"
```

```text nocopy theme={null}
Below 610.57.04: continue to step 2
```

If it prints `nothing to do`, stop here.

## 2. Prepare the build and the repository

Each command in this step installs only what is missing.

Kernel headers for the running kernel and for kernels installed later. DKMS
needs them to build the driver:

```bash theme={null}
sudo apt-get install linux-headers-$(uname -r) $(dpkg-query -W -f='${db:Status-Abbrev} ${Package} ${source:Package}\n' 'linux-image-*' 2>/dev/null | awk '$1 ~ /^[hi]i/ && $3 ~ /^linux-meta/ { sub(/^linux-image-(extra-)?/, "linux-headers-", $2); print $2 }' | sort -u)
```

The compilers your kernels were built with. DKMS builds with the same one, and
on Ubuntu 22.04 the HWE kernel needs GCC 12, which is not installed by default:

```bash theme={null}
sudo apt-get install $(sed -n 's|^CONFIG_CC_VERSION_TEXT="*[^" ]*-\(gcc-[0-9]*\) .*|\1|p' /lib/modules/*/build/.config | sort -u)
```

NVIDIA's repository
([NVIDIA driver installation guide](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/ubuntu.html)):

```bash theme={null}
. /etc/os-release; ARCH=$(uname -m | sed 's/aarch64/sbsa/'); wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${VERSION_ID//./}/$ARCH/cuda-keyring_1.1-1_all.deb
```

```bash theme={null}
sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt-get update
```

`apt-get update` must finish without `E:` lines. For a `Signed-By` error, see
[If apt-get update reports a Signed-By conflict](#if-apt-get-update-reports-a-signed-by-conflict).

Confirm NVIDIA publishes the version for this machine:

```bash theme={null}
apt-cache policy nvidia-driver-pinning-$DRIVER_VERSION | grep Candidate
```

```text nocopy theme={null}
  Candidate: 610.57.04-1ubuntu1
```

If it prints nothing, stop: nothing has been removed yet, and the machine keeps
its current driver.

## 3. Remove the current driver

This removes a driver installed with NVIDIA's `.run` installer, and does nothing
otherwise:

```bash theme={null}
command -v nvidia-installer && sudo nvidia-installer --uninstall
```

List the installed driver packages from Ubuntu or NVIDIA, including prebuilt
modules and the pin for a previous version. The NVIDIA container toolkit is not
in the list:

```bash theme={null}
OLD_DRIVER_PACKAGES=$(dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${source:Package}\n' | awk -F'\t' '$1 ~ /^[hi]i/ && $3 ~ /^(nvidia-graphics-drivers|nvidia-kmod-open|nvidia-open|cuda-drivers|linux-restricted-(modules|signatures)|nvidia-settings|nvidia-persistenced|nvidia-modprobe|nvidia-xconfig|(nvidia-)?fabric-?manager|libnvidia-nscq|libnvsdm|(nvidia-)?imex|nvlink5|nvidia-driver-pinning)/ { print $2 }'); echo "$OLD_DRIVER_PACKAGES"
```

If a name in the list contains `fabricmanager`, the machine has NVSwitch; step 4
installs Fabric Manager again. Remove them:

```bash theme={null}
sudo apt-get purge --allow-change-held-packages $OLD_DRIVER_PACKAGES
```

```text nocopy theme={null}
The following packages will be REMOVED:
  libnvidia-cfg1-580* libnvidia-common-580* libnvidia-compute-580*
  ...
  linux-modules-nvidia-580-open-generic* nvidia-compute-utils-580*
  nvidia-driver-580-open* nvidia-firmware-580-580.178.04*
  ...
The following held packages will be changed:
  nvidia-driver-580-open
Do you want to continue? [Y/n]
```

Before you answer, read the `REMOVED` list. Every name must contain `nvidia`,
`cuda`, `nvlink` or `nvsdm`, or be `libxnvctrl0`, and the list must not include
`nvidia-container-toolkit`,
`nvidia-container-toolkit-base`, `libnvidia-container1` or
`libnvidia-container-tools`. If either rule is broken, answer `n` and stop.

## 4. Install the driver

Pin the version first. NVIDIA's pinning package makes apt install exactly that
version of every driver package
([NVIDIA: version locking](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/version-locking.html)):

```bash theme={null}
sudo apt-get install nvidia-driver-pinning-$DRIVER_VERSION
```

Then the driver:

```bash theme={null}
sudo apt-get install $DRIVER_PACKAGE
```

On NVSwitch machines (HGX and DGX), install the NVSwitch components in the same
command instead
([NVIDIA: optional components](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/optional-components.html)).
Fabric Manager must match the driver version
([Fabric Manager user guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html)),
and the pin keeps all of them there. On ARM64, leave out `libnvsdm`, which
NVIDIA publishes for x86\_64 only:

```bash theme={null}
sudo apt-get install $DRIVER_PACKAGE nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex
```

DKMS builds the module during the install:

```text nocopy theme={null}
Loading new nvidia/610.57.04 DKMS files...
Building for 6.8.0-147-generic
Building initial module nvidia/610.57.04 for 6.8.0-147-generic
Installing /lib/modules/6.8.0-147-generic/updates/dkms/nvidia.ko.zst
```

If the output contains `Error!`, see
[If the DKMS build fails](#if-the-dkms-build-fails).

Check the module the reboot will load. A reboot normally starts the newest
kernel in `/boot`, and this prints the driver version built for it:

```bash theme={null}
modinfo -k $(ls /boot/vmlinuz-* | sed 's|.*/vmlinuz-||' | sort -V | tail -1) -F version nvidia
```

```text nocopy theme={null}
610.57.04
```

It must print your `DRIVER_VERSION`. `modinfo: ERROR: Module nvidia not found.`
means DKMS did not build for that kernel; see
[If the DKMS build fails](#if-the-dkms-build-fails). Do not reboot until it
prints the version.

## 5. Hold the driver

```bash theme={null}
sudo apt-mark hold $(dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${source:Package}\n' | awk -F'\t' '$1 ~ /^[hi]i/ && $3 ~ /^(nvidia-graphics-drivers|nvidia-kmod-open|nvidia-open|cuda-drivers|nvidia-settings|nvidia-persistenced|nvidia-modprobe|nvidia-xconfig|(nvidia-)?fabric-?manager|libnvidia-nscq|libnvsdm|(nvidia-)?imex|nvlink5|nvidia-driver-pinning)/ { print $2 }')
```

```text nocopy theme={null}
libnvidia-cfg1 set on hold.
libnvidia-compute set on hold.
...
nvidia-driver-pinning-610.57.04 set on hold.
nvidia-open set on hold.
...
```

## 6. Reboot and verify

```bash theme={null}
sudo reboot
```

If the machine does not come back, see
[If the machine does not come back](https://docs.vast.ai/host/upgrade-kernel.md#if-the-machine-does-not-come-back).

```bash theme={null}
nvidia-smi --query-gpu=name,driver_version --format=csv,noheader
```

```text nocopy theme={null}
NVIDIA GeForce RTX 4090, 610.57.04
NVIDIA GeForce RTX 4090, 610.57.04
```

Every line must show the minimum you set in step 1.

> [!TIP]
>
> Every GPU on the minimum driver version, module built by DKMS, driver held.

***

## Kernel upgrades

When a kernel upgrade installs new headers, DKMS builds the held driver for the
new kernel as part of the upgrade. Before rebooting into the new kernel, check:

```bash theme={null}
dkms status nvidia
```

```text nocopy theme={null}
nvidia/610.57.04, 6.8.0-147-generic, x86_64: installed
nvidia/610.57.04, 7.0.0-38-generic, x86_64: installed
```

The new kernel must have a line ending in `installed`. If it has none, do not
reboot. Run the two install commands at the start of step 2, then build for the
new kernel, using its version in place of `7.0.0-38-generic`:

```bash theme={null}
sudo dkms autoinstall -k 7.0.0-38-generic
```

***

## Recovery

### If the DKMS build fails

When the kernel's headers are missing, DKMS says so:

```text nocopy theme={null}
Error! Your kernel headers for kernel 7.0.0-38-generic cannot be found at /lib/modules/7.0.0-38-generic/build or /lib/modules/7.0.0-38-generic/source.
```

Otherwise the build log has the cause:

```bash theme={null}
grep -m5 error /var/lib/dkms/nvidia/$DRIVER_VERSION/build/make.log
```

```text nocopy theme={null}
cc: error: unrecognized command-line option '-ftrivial-auto-var-init=zero'
```

That line means the kernel's compiler is missing. For either cause, run the two
install commands at the start of step 2. Once they install what was missing,
apt finishes the interrupted driver install and builds the module. Then repeat
the `modinfo` check at the end of step 4.

### If nvidia-smi fails after the reboot

`NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`
means no module loaded for the running kernel:

```bash theme={null}
uname -r; dkms status nvidia
```

If no line for the kernel printed first ends in `installed`, build it and
reboot:

```bash theme={null}
sudo dkms autoinstall -k $(uname -r)
```

Until it is fixed the machine shows as offline; see
[Offline Machine](https://docs.vast.ai/host/machine-offline.md).

### If apt-get update reports a Signed-By conflict

```text nocopy theme={null}
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != /usr/share/keyrings/nvidia-old-keyring.gpg
```

NVIDIA's repository was already configured with another key file. Find both
definitions:

```bash theme={null}
grep -rl developer.download.nvidia.com/compute/cuda /etc/apt/sources.list /etc/apt/sources.list.d/
```

Keep `cuda-ubuntu2404-x86_64.list` (`ubuntu2204` on 22.04, `sbsa` on ARM64),
which `cuda-keyring` installed. Move the other file out of
`/etc/apt/sources.list.d/`, or comment out the line if it is in
`/etc/apt/sources.list`, then run `sudo apt-get update` again.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.