12 November 2026 · VRAMfarm team

Fine-tune an LLM: how much VRAM and how many GPUs for QLoRA, LoRA and full

Most fine-tuning jobs need one GPU, not eight. QLoRA trains an 8B model on a 16 GB card, a 32B model on a 48 GB card and a 70B model on one 80 GB card. LoRA in 16-bit needs two to three times as much. Only full fine-tuning, which updates every weight, pushes you to several 80 GB cards even for an 8B model, because it costs about 16 bytes per parameter before activations. A typical QLoRA run on an 8B model with 15 million training tokens takes 2–4 hours on an RTX 4090 and costs €1.65 in GPU time.

First: do you need to fine-tune?

Fine-tuning teaches a model a format, a tone or a narrow task: answering in your support style, extracting fields into your JSON schema, classifying tickets into your categories. It is a poor way to add knowledge. If the model should know your product documentation, put the documents in front of it at question time (retrieval, or simply a long prompt) and keep them up to date without retraining. Try a strong prompt with a few examples on the base model first; when it still gets the format or the style wrong in one case out of five, a fine-tune is worth it.

Memory per method: the arithmetic

Three methods, three very different memory bills:

• QLoRA loads the base model in 4-bit (about 0.6 bytes per parameter) and trains small adapter matrices next to it. The base stays frozen, so there are no gradients or optimiser states for it.
• LoRA does the same with the base in 16-bit: 2 bytes per parameter, slightly better quality, more memory.
• Full fine-tuning updates every weight. With the standard AdamW optimiser in mixed precision, each parameter costs about 16 bytes: 16-bit weights and gradients, an fp32 master copy and two fp32 optimiser states. An 8-bit optimiser and 16-bit weights without the master copy bring it to about 6–8 bytes.

On top come activations, which grow with sequence length and batch size. The figures below assume 2,000–4,000-token sequences, a small batch and gradient checkpointing, which recomputes activations instead of storing them.

Ranges, because trainers differ: Unsloth’s kernels need noticeably less than plain Hugging Face code, and long sequences push every row up. The VRAM guide explains the bytes-per-parameter arithmetic, and the CUDA out of memory guide the knobs to turn when a run doesn’t fit: batch size, sequence length, gradient checkpointing, 8-bit optimisers.

Full fine-tuning is rarely what a task needs. LoRA on all linear layers with a rank of 16–64 recovers most of the gain for style, format and narrow tasks. Full fine-tuning earns its cost for continued pre-training on a large corpus, a new language or a deep domain shift.

One card or several

Two different things happen when you add cards, and only one of them saves memory.

Data parallel (DDP): every card holds a full copy of the model and processes different batches. Memory per card stays the same; speed grows, roughly 1.8 times with two cards and 3.5 times with four. Useful when a run fits on one card but would take all night.

Sharding (FSDP, DeepSpeed ZeRO-3): weights, gradients and optimiser states are split across the cards, so together they hold a model none of them could alone. This is how full fine-tuning and 16-bit LoRA of large models work. The cards exchange data at every step, so the connection matters: data-centre machines with NVLink, such as 8 × H100, scale well; consumer cards like the RTX 4090 and 5090 talk over PCIe and scale noticeably worse. QLoRA combined with FSDP can even split a 70B model over two 24 GB cards, at a modest speed.

The practical rule: if the job fits on one card, use one card. It is simpler and nothing can go wrong between cards. A single H100 often finishes sooner and cheaper than two consumer cards. Take several cards when the model doesn’t fit any single card, or when a one-card run would take more than a working day. Machines with 2, 4 and 8 cards are listed separately in the catalogue, and multi-GPU training basics covers the setup; a multi-card machine costs about the number of cards times the single-card price, so 4 × H100 is about €15.60 an hour.

Tools: Unsloth, Axolotl, TRL

Unsloth is the quickest way to QLoRA or LoRA on one card. Its custom kernels cut memory and time, it ships ready 4-bit versions of popular models, and it exports the result as an adapter, a merged 16-bit model for vLLM or a GGUF for Ollama. Multi-GPU support is newer and more limited than in the other two.

pip install unsloth

If pip replaces the image’s PyTorch on the way, check that the new build matches the CUDA version the machine showed before launch; python -c "import torch; print(torch.cuda.is_available())" should print True.

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-8B", max_seq_length=2048, load_in_4bit=True)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth")

data = load_dataset("json", data_files="/workspace/data/train.jsonl", split="train")
trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=data,
    args=SFTConfig(output_dir="/workspace/out", per_device_train_batch_size=2,
        gradient_accumulation_steps=4, num_train_epochs=3, learning_rate=2e-4,
        logging_steps=10, save_steps=200, bf16=True))
trainer.train()
model.save_pretrained("/workspace/out/adapter")

train.jsonl holds one example per line, each a messages list in chat format (system, user, assistant), which the trainer formats with the model’s own chat template.

Axolotl describes the whole run in one YAML file: model, dataset, LoRA or full, sequence length, and how to use several GPUs. That makes runs repeatable and easy to compare. Install it with pip as its documentation describes (the extras for flash-attention and DeepSpeed depend on your setup), then axolotl fetch examples downloads ready configs for common models, and axolotl train your-config.yaml starts the run. For a model that doesn’t fit one card, add a DeepSpeed ZeRO-3 or FSDP section to the config; this is where Axolotl is stronger than Unsloth.

TRL from Hugging Face is the library under much of the above: supervised fine-tuning, DPO and GRPO trainers, PEFT for LoRA. Choose it when you want your own training code. A script that works on one card runs on four with accelerate launch --num_processes 4 train.py.

Time and cost

Planning ranges for one dataset: 10,000 examples of about 500 tokens, three epochs, 15 million training tokens. Throughput comes from compute arithmetic and public community reports, not our own benchmarks. Time scales with tokens, so double the data, double the hours.

Two patterns stand out. Faster data-centre cards often cost less per finished run, because the hours shrink faster than the price grows: compare the two 70B rows. And the full fine-tune of an 8B model is short in GPU time; what takes longer is preparing the data and the evaluation around it.

Add the setup on top: 5–10 minutes to install the trainer, and the model download, which is billed traffic plus minutes. Take the 4-bit version for QLoRA where one exists; for a 70B model that is about 40 GB instead of 141 GB. The cost per million tokens article uses the same method for running models after training.

Running it on a rented server without losing work

1. Test small first. Train on 1% of the data in the first 15 minutes, which you pay for anyway. Note the tokens per second the trainer prints and the memory in nvidia-smi, then multiply out before starting the full run.
2. Keep it running without you. Start the training inside tmux (apt-get install -y tmux, tmux new -s train) or with nohup python train.py > /workspace/train.log 2>&1 &. Closing the browser or the SSH session then doesn’t stop it.
3. Checkpoint and copy off. Set save_steps so a checkpoint lands every 15–30 minutes in /workspace/out. Hardware fails now and then; if a machine goes missing, billing stops from that moment, but its disk goes with it. Copy checkpoints off as you go: a private Hugging Face repository, rsync to your own computer or your own bucket. The article on files after a stop explains what the saved folder keeps.
4. Keep the adapter, not the base. A LoRA adapter is tens to a few hundred MB, so it costs next to nothing in the saved folder at €0.03 per GB per month. The public base model can be downloaded again.
5. Stop when done. Billing stops the minute you stop the server. Idle protection emails you after 2 hours without SSH, Jupyter or GPU load, and an hour later saves the folder and stops the server, but those idle hours are billed.

To use the result, serve the base model with the adapter in vLLM: vllm serve Qwen/Qwen3-8B --enable-lora --lora-modules support=/workspace/out/adapter, then call the model name support through its OpenAI-compatible API. Or merge the adapter into the base with Unsloth and export a GGUF for Ollama. The runtime comparison helps you pick which.