Upgrade the NVIDIA Driver

Before you start: schedule a maintenance window at least 48 hours in advance.You are responsible for the maintenance and any resulting interruption. Work performed outside the scheduled maintenance window may be treated as an operational failure.If you are unsure whether this operation could cause data loss, stop here.

Vast asks hosts to run at least a minimum NVIDIA driver version, set by GPU generation and, on Turing and newer, by the driver the machine runs now. When Vast raises a minimum, this table is updated; follow this page again to move to it. The steps are the same every time.

GPUs	Compute capability	Current driver	Minimum driver version
Turing and newer	7.5 and higher	595.xx	595.91.07
Turing and newer	7.5 and higher	Any other	610.57.04
Maxwell, Pascal and Volta	below 7.5	Any	580.178.04

A machine at or above its minimum needs nothing from this page. For a machine below it, this page installs the minimum version from NVIDIA’s apt repository, has DKMS build its kernel module, and holds it so nothing upgrades it. With DKMS, a kernel update rebuilds the same driver for the new kernel instead of changing the driver.

Driver branches newer than the ones in the table, such as 620, are not directly supported yet, though a machine on one may still run on the platform. Hosts on an unlisted branch are expected to keep the driver on the newest release of that branch, for its latest security fixes. This page does not change it.

Before step 3, unlist the machine and make sure it has no instances; a maintenance window does neither. From step 3 until the reboot in step 6 the GPUs are unusable, and the reboot stops every instance.

1. Check the current driver

nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv,noheader

NVIDIA GeForce RTX 4090, 8.9, 580.178.04
NVIDIA GeForce RTX 4090, 8.9, 580.178.04

The second column is the compute capability and the third is the current driver. Set the minimum from the table row that matches both; it is also the version this page installs. The later commands read these two variables, so if you log out, set them again before you continue. Compute capability 7.5 or higher, current driver 595.xx:

DRIVER_VERSION=595.91.07; DRIVER_PACKAGE=nvidia-open

Compute capability 7.5 or higher, any other driver:

DRIVER_VERSION=610.57.04; DRIVER_PACKAGE=nvidia-open

Below 7.5:

DRIVER_VERSION=580.178.04; DRIVER_PACKAGE=nvidia-driver-580

Check whether the machine already meets the minimum:

dpkg --compare-versions "$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | sort -u)" ge "$DRIVER_VERSION" && echo "At or above $DRIVER_VERSION: nothing to do" || echo "Below $DRIVER_VERSION: continue to step 2"

Below 610.57.04: continue to step 2

If it prints nothing to do, stop here.

2. Prepare the build and the repository

Each command in this step installs only what is missing. Kernel headers for the running kernel and for kernels installed later. DKMS needs them to build the driver:

sudo apt-get install linux-headers-$(uname -r) $(dpkg-query -W -f='${db:Status-Abbrev} ${Package} ${source:Package}\n' 'linux-image-*' 2>/dev/null | awk '$1 ~ /^[hi]i/ && $3 ~ /^linux-meta/ { sub(/^linux-image-(extra-)?/, "linux-headers-", $2); print $2 }' | sort -u)

The compilers your kernels were built with. DKMS builds with the same one, and on Ubuntu 22.04 the HWE kernel needs GCC 12, which is not installed by default:

sudo apt-get install $(sed -n 's|^CONFIG_CC_VERSION_TEXT="*[^" ]*-\(gcc-[0-9]*\) .*|\1|p' /lib/modules/*/build/.config | sort -u)

NVIDIA’s repository (NVIDIA driver installation guide):

. /etc/os-release; ARCH=$(uname -m | sed 's/aarch64/sbsa/'); wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${VERSION_ID//./}/$ARCH/cuda-keyring_1.1-1_all.deb

sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt-get update

apt-get update must finish without E: lines. For a Signed-By error, see If apt-get update reports a Signed-By conflict. Confirm NVIDIA publishes the version for this machine:

apt-cache policy nvidia-driver-pinning-$DRIVER_VERSION | grep Candidate

  Candidate: 610.57.04-1ubuntu1

If it prints nothing, stop: nothing has been removed yet, and the machine keeps its current driver.

3. Remove the current driver

This removes a driver installed with NVIDIA’s .run installer, and does nothing otherwise:

command -v nvidia-installer && sudo nvidia-installer --uninstall

List the installed driver packages from Ubuntu or NVIDIA, including prebuilt modules and the pin for a previous version. The NVIDIA container toolkit is not in the list:

OLD_DRIVER_PACKAGES=$(dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${source:Package}\n' | awk -F'\t' '$1 ~ /^[hi]i/ && $3 ~ /^(nvidia-graphics-drivers|nvidia-kmod-open|nvidia-open|cuda-drivers|linux-restricted-(modules|signatures)|nvidia-settings|nvidia-persistenced|nvidia-modprobe|nvidia-xconfig|(nvidia-)?fabric-?manager|libnvidia-nscq|libnvsdm|(nvidia-)?imex|nvlink5|nvidia-driver-pinning)/ { print $2 }'); echo "$OLD_DRIVER_PACKAGES"

If a name in the list contains fabricmanager, the machine has NVSwitch; step 4 installs Fabric Manager again. Remove them:

sudo apt-get purge --allow-change-held-packages $OLD_DRIVER_PACKAGES

The following packages will be REMOVED:
  libnvidia-cfg1-580* libnvidia-common-580* libnvidia-compute-580*
  ...
  linux-modules-nvidia-580-open-generic* nvidia-compute-utils-580*
  nvidia-driver-580-open* nvidia-firmware-580-580.178.04*
  ...
The following held packages will be changed:
  nvidia-driver-580-open
Do you want to continue? [Y/n]

Before you answer, read the REMOVED list. Every name must contain nvidia, cuda, nvlink or nvsdm, or be libxnvctrl0, and the list must not include nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container1 or libnvidia-container-tools. If either rule is broken, answer n and stop.

4. Install the driver

Pin the version first. NVIDIA’s pinning package makes apt install exactly that version of every driver package (NVIDIA: version locking):

sudo apt-get install nvidia-driver-pinning-$DRIVER_VERSION

Then the driver:

sudo apt-get install $DRIVER_PACKAGE

On NVSwitch machines (HGX and DGX), install the NVSwitch components in the same command instead (NVIDIA: optional components). Fabric Manager must match the driver version (Fabric Manager user guide), and the pin keeps all of them there. On ARM64, leave out libnvsdm, which NVIDIA publishes for x86_64 only:

sudo apt-get install $DRIVER_PACKAGE nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex

DKMS builds the module during the install:

Loading new nvidia/610.57.04 DKMS files...
Building for 6.8.0-147-generic
Building initial module nvidia/610.57.04 for 6.8.0-147-generic
Installing /lib/modules/6.8.0-147-generic/updates/dkms/nvidia.ko.zst

If the output contains Error!, see If the DKMS build fails. Check the module the reboot will load. A reboot normally starts the newest kernel in /boot, and this prints the driver version built for it:

modinfo -k $(ls /boot/vmlinuz-* | sed 's|.*/vmlinuz-||' | sort -V | tail -1) -F version nvidia

610.57.04

It must print your DRIVER_VERSION. modinfo: ERROR: Module nvidia not found. means DKMS did not build for that kernel; see If the DKMS build fails. Do not reboot until it prints the version.

5. Hold the driver

sudo apt-mark hold $(dpkg-query -W -f='${db:Status-Abbrev}\t${binary:Package}\t${source:Package}\n' | awk -F'\t' '$1 ~ /^[hi]i/ && $3 ~ /^(nvidia-graphics-drivers|nvidia-kmod-open|nvidia-open|cuda-drivers|nvidia-settings|nvidia-persistenced|nvidia-modprobe|nvidia-xconfig|(nvidia-)?fabric-?manager|libnvidia-nscq|libnvsdm|(nvidia-)?imex|nvlink5|nvidia-driver-pinning)/ { print $2 }')

libnvidia-cfg1 set on hold.
libnvidia-compute set on hold.
...
nvidia-driver-pinning-610.57.04 set on hold.
nvidia-open set on hold.
...

6. Reboot and verify

sudo reboot

If the machine does not come back, see If the machine does not come back.

nvidia-smi --query-gpu=name,driver_version --format=csv,noheader

NVIDIA GeForce RTX 4090, 610.57.04
NVIDIA GeForce RTX 4090, 610.57.04

Every line must show the minimum you set in step 1.

Every GPU on the minimum driver version, module built by DKMS, driver held.

Kernel upgrades

When a kernel upgrade installs new headers, DKMS builds the held driver for the new kernel as part of the upgrade. Before rebooting into the new kernel, check:

dkms status nvidia

nvidia/610.57.04, 6.8.0-147-generic, x86_64: installed
nvidia/610.57.04, 7.0.0-38-generic, x86_64: installed

The new kernel must have a line ending in installed. If it has none, do not reboot. Run the two install commands at the start of step 2, then build for the new kernel, using its version in place of 7.0.0-38-generic:

sudo dkms autoinstall -k 7.0.0-38-generic

Recovery

If the DKMS build fails

When the kernel’s headers are missing, DKMS says so:

Error! Your kernel headers for kernel 7.0.0-38-generic cannot be found at /lib/modules/7.0.0-38-generic/build or /lib/modules/7.0.0-38-generic/source.

Otherwise the build log has the cause:

grep -m5 error /var/lib/dkms/nvidia/$DRIVER_VERSION/build/make.log

cc: error: unrecognized command-line option '-ftrivial-auto-var-init=zero'

That line means the kernel’s compiler is missing. For either cause, run the two install commands at the start of step 2. Once they install what was missing, apt finishes the interrupted driver install and builds the module. Then repeat the modinfo check at the end of step 4.

If nvidia-smi fails after the reboot

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver means no module loaded for the running kernel:

uname -r; dkms status nvidia

If no line for the kernel printed first ends in installed, build it and reboot:

sudo dkms autoinstall -k $(uname -r)

Until it is fixed the machine shows as offline; see Offline Machine.

If apt-get update reports a Signed-By conflict

E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != /usr/share/keyrings/nvidia-old-keyring.gpg

NVIDIA’s repository was already configured with another key file. Find both definitions:

grep -rl developer.download.nvidia.com/compute/cuda /etc/apt/sources.list /etc/apt/sources.list.d/

Keep cuda-ubuntu2404-x86_64.list (ubuntu2204 on 22.04, sbsa on ARM64), which cuda-keyring installed. Move the other file out of /etc/apt/sources.list.d/, or comment out the line if it is in /etc/apt/sources.list, then run sudo apt-get update again.

Concepts

Guides

CLI

SDK

1. Check the current driver

2. Prepare the build and the repository

3. Remove the current driver

4. Install the driver

5. Hold the driver

6. Reboot and verify

Kernel upgrades

Recovery

If the DKMS build fails

If nvidia-smi fails after the reboot

If apt-get update reports a Signed-By conflict

​1. Check the current driver

​2. Prepare the build and the repository

​3. Remove the current driver

​4. Install the driver

​5. Hold the driver

​6. Reboot and verify

​Kernel upgrades

​Recovery

​If the DKMS build fails

​If nvidia-smi fails after the reboot

​If apt-get update reports a Signed-By conflict

1. Check the current driver

2. Prepare the build and the repository

3. Remove the current driver

4. Install the driver

5. Hold the driver

6. Reboot and verify

Kernel upgrades

Recovery

If the DKMS build fails

If nvidia-smi fails after the reboot

If apt-get update reports a Signed-By conflict