11 November 2026 · VRAMfarm team

CUDA driver version mismatch: fix PyTorch and ComfyUI on a rented GPU

You launch a server, install your project, and the first check prints torch.cuda.is_available() False. Or ComfyUI stops with “The NVIDIA driver on your system is too old”. On a rented machine several version numbers are in play, and the one that decides is usually the PyTorch build, which you can change. The driver belongs to the host. So the fix is nearly always one of two things: install the PyTorch wheel that matches the driver, or pick a machine whose driver matches the build you need. Both take a few minutes once you know which number is which.

Driver, CUDA toolkit and PyTorch build: three different numbers

One rule covers most cases: the CUDA version of your PyTorch build must not be newer than the “CUDA Version” nvidia-smi shows. Drivers are backward compatible. A driver that supports CUDA 12.8 runs cu118, cu124, cu126 and cu128 builds. A cu130 build won’t run on it, and a cu129 build may or may not (see the note under the table).

Minimum Linux driver for each CUDA build:

CUDA 12.x programs can sometimes run on an older 12-series driver (525 or newer) through what NVIDIA calls minor-version compatibility, but some features need the full driver and the failures show up in odd places. Don’t plan around it.

Check the machine in under a minute

Open the Jupyter terminal or connect over SSH (how to set up an SSH key) and run:

nvidia-smi
nvcc --version
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available()); print(torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0)); print(torch.cuda.get_arch_list())"

How to read the result:

• nvidia-smi gives the driver and the newest CUDA it supports. If nvidia-smi itself fails, the problem is on the host, not in your software (see below).
• nvcc may say “command not found”. That’s fine for PyTorch, which doesn’t need it.
• The Python line should print something like 2.8.0+cu128 12.8 True, then the card’s name and its compute capability, such as (12, 0) for an RTX 5090. The architecture list must include your card: sm_80 for the A100, sm_86 for RTX 30-series and RTX A4000–A6000, sm_89 for RTX 40-series, L4, L40 and L40S, sm_90 for H100 and H200, sm_120 for RTX 50-series and RTX PRO 6000, sm_100 for B200.

If torch.version.cuda prints None, you have a CPU-only build. ComfyUI prints the same facts when it starts: look for the “pytorch version” line with its +cu suffix and the “Device: cuda:0” line in the log.

What each error means and how to fix it

“RuntimeError: No CUDA GPUs are available”

Also shows up as torch.cuda.is_available() returning False. Work through the causes in order:

1. echo $CUDA_VISIBLE_DEVICES prints an empty value or a wrong index. Unset it, or set it to 0.
2. torch.version.cuda is None: a CPU build. Reinstall as shown in the next section.
3. The PyTorch build is newer than the driver supports, and the driver check failed quietly. Compare the two numbers.
4. nvidia-smi fails too: the container has no access to the GPU. That’s a host problem.

“The NVIDIA driver on your system is too old (found version 12020)”

The number is the newest CUDA the driver supports, written without dots: 12020 is 12.2, 12080 is 12.8. Your PyTorch build needs newer. Either install an older build that fits, which for a 12.2 driver means an old PyTorch release, or move to a machine with a newer driver. For current projects the second option is usually faster.

“AssertionError: Torch not compiled with CUDA enabled”

A CPU-only build is installed. On Linux it typically came from the CPU index, a +cpu version pinned in a requirements file, or a package that reinstalled torch as its own dependency. ComfyUI custom nodes do this more often than you’d expect.

“CUDA error: no kernel image is available for execution on the device”

Usually preceded by a warning such as “NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation”. The build has no code for your card’s architecture. On Blackwell cards this means PyTorch older than 2.7 or a build older than cu128. The same error comes from extensions compiled without your architecture: xformers, flash-attn, older bitsandbytes or attention kernels pinned to an old version.

“nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.8”

This one appears with your own Docker image. NVIDIA’s CUDA base images declare the CUDA version they need, and the container won’t start on a machine whose driver is older. Rebuild on a base image with a lower CUDA version, or choose a machine with a newer driver.

“Failed to initialize NVML: Driver/library version mismatch”

If nvidia-smi fails with this or with “couldn’t communicate with the NVIDIA driver”, the host’s driver is broken or changed under the container. You can’t fix it from inside. Save /workspace and stop the server (billing ends once it has stopped), start another one, and tell support which machine it was.

“ImportError: libcudnn.so.9: cannot open shared object file”

Two packages expect different CUDA libraries, for example PyTorch for CUDA 12 and an ONNX Runtime or TensorFlow build for another version. Run pip check, keep everything on one CUDA generation, and when in doubt start a fresh virtual environment.

Install the PyTorch build that matches

1. Read “CUDA Version” in nvidia-smi.
2. Pick the newest PyTorch build that is not newer: 12.8 → cu128; 12.6 → cu126; 13.0 → cu130 or cu128. The selector on pytorch.org lists the builds the current release offers.
3. Install it in a virtual environment, with torch, torchvision and torchaudio together from the same index so their versions match:

python -m venv /workspace/venv
source /workspace/venv/bin/activate
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

Keeping the environment in /workspace means it goes into the saved folder when you stop. Remember that the next machine then needs a driver new enough for that build too.

4. Stop other packages from replacing it. Write the installed version into a constraints file and make pip respect it:

pip freeze | grep -E "^torch(vision|audio)?==" > /workspace/constraints.txt
export PIP_CONSTRAINT=/workspace/constraints.txt

5. Install extensions for the same build. xformers publishes wheels on the same PyTorch index; a plain pip install xformers can pull in a different torch. flash-attn has prebuilt wheels for common torch and CUDA pairs. Compiling it yourself needs the CUDA toolkit with the same major version and can take an hour or more of billed time, depending on the CPU.

For ComfyUI: after installing custom nodes through the Manager, run the Python check again. If a node swapped torch, reinstall it with the line above, then pip install -r requirements.txt in the ComfyUI folder. The ComfyUI template comes with PyTorch on CUDA 12.8.

RTX 50-series and other Blackwell cards need CUDA 12.8 or newer

The RTX 5090, RTX 5080, RTX 5070 Ti and RTX PRO 6000 use Blackwell’s sm_120 architecture; the B200 uses sm_100. They need all three of these:

• driver 570 or newer;
• a PyTorch build for CUDA 12.8 or later (cu128, cu129, cu130);
• PyTorch 2.7 or newer, the first stable release with Blackwell support.

Projects pinned to an old stack, such as torch 2.1 with cu118 in an older training repo, won’t run on Blackwell at all. Ampere and Ada cards still run those builds, so for an old project an RTX 4090 or RTX 3090 is the practical choice. Most popular extensions now publish wheels with sm_120; older pinned versions of them don’t. If an extension fails on an RTX 5090, check that its version is recent before suspecting the card.

You can’t update the driver, so choose it before launch

The driver is part of the host’s operating system. Your server is a container that gets the driver’s libraries passed in from outside. Installing a driver package inside it changes nothing useful and can break those libraries. NVIDIA’s forward-compatibility package helps only on data-centre cards and isn’t a general fix.

VRAMfarm shows each machine’s driver and CUDA version before you launch it, in the launch wizard in the dashboard and in the server catalogue, where you can filter by minimum CUDA version. Pick it the same way you pick memory:

• a stack on cu128 → CUDA 12.8 or newer, driver 570 or newer;
• cu130 → CUDA 13.0 or newer, driver 580 or newer;
• an old project on cu118 → any current driver, but not a Blackwell card.

For your own Docker image, build on the lowest CUDA version your stack really needs, not the newest. An image on CUDA 12.6 starts on more machines than one on 13.0. If a version error remains after these checks, write to support with the error text and the output of the three commands above; that is usually enough to find the cause. Other common questions about servers are answered in the FAQ.