11 November 2026 · VRAMfarm team

CUDA out of memory: how to fix it in ComfyUI, LLMs and training

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 23.65 GiB of which 1.12 GiB is free.
Of the allocated memory 20.10 GiB is allocated by PyTorch, and 1.31 GiB
is reserved by PyTorch but unallocated.

That message says one thing: at that moment the program asked for 2 GiB more video memory than was left. The card is fine, and it is not always too small. Most cases are fixed in minutes with one of four levers: lower precision, a smaller unit of work (batch, resolution, context), freeing memory that something else holds, or a fix for fragmentation. When none of them works without wrecking quality or speed, a bigger card is the cheaper answer.

Read the error before changing anything

Each part of the message tells you something:

• “Tried to allocate 2.00 GiB” is the single request that failed, not the total your job needs. Lowering settings slightly may only move the crash a few steps later.
• “Total capacity of 23.65 GiB” on a 24 GB card is normal. The driver keeps a small part of the memory for itself, and the CUDA context takes a few hundred megabytes more before your model loads.
• “Reserved by PyTorch but unallocated” is memory PyTorch holds but can’t use for this request. If it runs to gigabytes, the problem is fragmentation; see below.
• “Process 1234 has 9.80 GiB memory in use” lines, when present, mean another process shares the card.

RuntimeError: CUDA error: out of memory without the details usually comes from a library such as cuBLAS or a custom kernel, often during start-up. The fixes are the same.

Check what holds the memory with nvidia-smi

Run nvidia-smi in the Jupyter terminal or over SSH. The line that matters is per GPU:

| 0  NVIDIA GeForce RTX 4090  ...  | 21540MiB / 24564MiB |  97%  Default |

The first figure is memory in use, the second the card’s total, then GPU load. To watch it live while your job runs:

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 1

On a rented server the process list at the bottom is often empty, even when memory is in use. The server is a container, and nvidia-smi can’t see process names across that boundary. Use ps aux | grep python instead.

Three things commonly hold memory you didn’t expect:

1. PyTorch’s cache. Freed tensors stay reserved for reuse, so nvidia-smi shows them as used. torch.cuda.memory_allocated() shows what is really in use, torch.cuda.memory_reserved() what is held. torch.cuda.empty_cache() hands the cache back, which helps only when another process needs it.
2. Another process on the same card. A Jupyter kernel that loaded a model three cells ago, an old ComfyUI still running, a training script that crashed half-way. Restart the kernel or kill the PID. If your script dies every time another app starts on the card, this is why: both compete for the same memory.
3. vLLM’s reservation. vLLM takes 90% of the card at start by default. Anything else you start next to it will run out.

ComfyUI, Flux and Wan: fixes in order

Work down the list and stop when it runs:

1. Use FP8 or GGUF weights. Flux.1 dev in 16-bit is about 24 GB of weights alone; in FP8, about 12 GB. Its T5-XXL text encoder drops from about 9.5 GB to about 5 GB in FP8. GGUF versions (through the ComfyUI-GGUF nodes) go further: Q8 suits 16 GB cards, Q5 and Q4 fit 12 GB, and at Q4 fine detail starts to suffer. Our Flux template already ships Flux.1 schnell in FP8.
2. Swap VAE Decode for VAE Decode (Tiled). Decoding at the end is a classic crash point, especially for large images and video. Tiled decoding works in pieces, at the cost of a few seconds.
3. Lower the batch size in Empty Latent Image. Four images one after another need about a quarter of the activation memory of a batch of four.
4. Lower the resolution, then upscale. Memory for activations grows with pixel count: 2048×2048 has four times the pixels of 1024×1024. Generate at native size and use a tiled upscaler.
5. For video, cut frames or resolution. With Wan 2.2, going from 81 to 49 frames or from 720p to 480p saves several gigabytes. The 14B model is two 14B experts that ComfyUI loads in turn; on 24–32 GB cards, block swap in the Wan video wrapper nodes moves part of the model to system RAM.
6. Offload as a last resort. When you start ComfyUI yourself, --lowvram keeps parts of the model in system RAM and --reserve-vram 2 leaves 2 GB free for other things. Expect generation to slow down several times.

The Manager’s buttons for unloading models and freeing the cache help when you switch between workflows in one session, for example from Flux to Wan, since the old model can stay loaded.

LLMs: weights plus the KV cache

A language model needs memory for three things: the weights, the KV cache and 1–2 GB of working overhead.

Weights are parameters times bytes per parameter:

The KV cache grows with every token in the context, for every request at once. In 16-bit it takes about 128 KB per token for Llama 3.1 8B, 256 KB for Qwen3 32B and 320 KB for Llama 3.x 70B. A single 32,000-token conversation on Qwen3 32B needs about 8 GB of cache; a 128,000-token one on a 70B model, about 42 GB, as much as the 4-bit weights. That is why a model loads fine and then fails on a long chat, or when ten users arrive together.

Fixes by tool:

• Ollama. Set a shorter context with num_ctx or OLLAMA_CONTEXT_LENGTH, and halve the cache with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0. Ollama rarely crashes; it moves layers to the CPU instead. If ollama ps shows something like 35%/65% CPU/GPU, the model doesn’t fit and runs several times slower.
• llama.cpp. Cap the context with -c 16384, choose how many layers stay on the GPU with -ngl, and quantise the cache with --cache-type-k q8_0 --cache-type-v q8_0. A quantised V cache needs flash attention: current builds turn it on automatically where the GPU supports it, and -fa on forces it.
• vLLM. --gpu-memory-utilization (default 0.9) is the share of the card’s whole memory vLLM takes for weights and cache at start. Lower it to 0.8 if start-up fails because something else uses the card. If vLLM says the model’s max seq len is larger than the KV cache can store, raise it to 0.95 or set --max-model-len 32768. --kv-cache-dtype fp8 halves the cache, --max-num-seqs caps parallel requests, --enforce-eager saves the memory CUDA graphs take at some cost to speed, and --tensor-parallel-size 2 splits the model across two cards. The vLLM template gives you an OpenAI-compatible API to try these settings.
• Quantise the weights. An AWQ or GPTQ 4-bit checkpoint cuts weights to about a quarter of 16-bit. FP8 halves them and runs natively on Ada, Hopper and Blackwell cards.

Training: shrink the batch, the activations and the optimiser

Full fine-tuning with AdamW in mixed precision needs about 16 bytes per parameter before activations: 2 for weights, 2 for gradients, 4 for a 32-bit master copy and 8 for the optimiser states. A 7B model is about 112 GB, more than any single card except the H200 and B200. Fixes, roughly in order of how much they save for how little they cost:

1. Smaller batch, same effective batch. Batch size 2 with 8 steps of gradient accumulation trains like a batch of 16.
2. Gradient checkpointing. Activations are recomputed during the backward pass instead of stored. It saves a large share of activation memory and costs 20–30% in speed. In Hugging Face: gradient_checkpointing=True.
3. BF16 mixed precision on any card from Ampere onwards.
4. 8-bit optimisers. bitsandbytes AdamW8bit (optim="adamw_bnb_8bit") cuts the optimiser states from 8 bytes per parameter to about 2.
5. LoRA or QLoRA instead of full fine-tuning. You train small adapters; QLoRA loads the base in 4-bit. An 8B model trains with QLoRA in 10–16 GB, a 70B model needs about 48 GB and is comfortable on 80 GB.
6. Shorter sequences or lower resolution. For a Flux LoRA, train at 512–768 px first and cache latents and text embeddings.
7. Several cards. FSDP or DeepSpeed ZeRO spread weights, gradients and optimiser states across GPUs.

If the crash comes after hundreds of steps rather than at once, look for a leak: total_loss += loss keeps the whole graph alive, total_loss += loss.item() doesn’t. Evaluation without torch.no_grad() does the same.

Fragmentation: PYTORCH_CUDA_ALLOC_CONF

When the error shows gigabytes “reserved but unallocated”, the memory is there but broken into pieces too small for the request. This happens most with changing shapes: different image sizes, sequence lengths that vary, or models loaded and unloaded in turn. Set this before Python starts:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

In a notebook, put os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True" in the first cell, before import torch. Newer PyTorch releases also read the same setting as PYTORCH_ALLOC_CONF. On older versions, max_split_size_mb:128 is the alternative. This setting won’t create memory; it only lets you use what is free.

To size your job, measure the peak instead of guessing: torch.cuda.max_memory_allocated() after one full step or one image, and torch.cuda.memory_summary() for the breakdown.

When to stop tuning and take a bigger card

Move up when the fix costs you quality (dropping from FP8 to Q4), speed (CPU offload) or more of your time than the price difference. For comparison, an hour costs from €0.59 on an RTX 4090 with 24 GB, €0.76 on an RTX 5090 with 32 GB, €1.06 on an L40S with 48 GB and €3.90 on an H100 with 80 GB.

The VRAM picker shows which cards fit your model and what they cost now; the guide to how much VRAM you need has the full table by model. A quick test on the bigger card costs as little as 15 minutes of its price, and if you save /workspace when you stop, it is restored on the next server even when that server has a different card.