12 November 2026 · VRAMfarm team

Run Llama 70B on one GPU: 4-bit or FP8, 48 GB or 80 GB, and real speeds

The first attempt usually goes like this: a 24 GB card, ollama run llama3.3:70b, and an answer that crawls out at two or three tokens per second. Nothing is broken. The 4-bit model is 43 GB, so Ollama quietly put almost half of it in system RAM. A single card runs a 70B model at reading speed only when the whole model fits: a 48 GB card for one person at normal context (about 12–18 tokens per second), or an 80 GB card for long documents, several users or FP8 quality (25–50 tokens per second).

Everything below uses Llama 3.3 70B, and the numbers hold for other dense 70B models built on it, such as DeepSeek R1 Distill Llama 70B. A 72B model such as Qwen2.5 72B is a few gigabytes bigger (about 47 GB in Q4_K_M), which leaves almost no cache room on a 48 GB card.

Pick the format first

Each runtime has its own file format, and the format decides the size, and with it the card.

GGUF is the llama.cpp format, and Ollama uses it too. One file holds the weights, and the runtime can keep some layers on the CPU if the GPU runs out, which is both its strength and the trap from the first paragraph. Q4_K_M is the usual choice for 70B.

AWQ and GPTQ are 4-bit formats for GPU servers. vLLM runs both with fast kernels on every card from the Ampere generation onwards, which covers everything in our catalogue. Their quality is similar; pick a build from a maintained source with plenty of downloads.

FP8 halves the original with almost no loss. The cards compute it natively from the Ada generation onwards (RTX 40 series, L40S), and on Hopper (H100, H200) and Blackwell (RTX 50 series, RTX PRO 6000, B200). A 70B model in FP8 needs about 71 GB, so in practice FP8 means an 80 GB H100, a 96 GB RTX PRO 6000 or an H200. The A100 can load FP8 weights in vLLM but computes in 16-bit, so it saves memory and gains no speed. Blackwell cards also compute NVFP4, a 4-bit format at about 40 GB that in published comparisons often stays closer to FP8 quality than older 4-bit methods; runtime support is newer, so check that your vLLM version lists it.

48 GB or 80 GB: the context decides

The weights are only part of the bill. Llama 3.3 70B keeps about 0.33 MB of KV cache per token in 16-bit, so 8,000 tokens of conversation take 2.7 GB and the full 128k context takes 43 GB, as much as the 4-bit weights. The VRAM guide explains the formula; here is what it means per card.

Ranges, because cards differ by a gigabyte or two in usable memory, every runtime keeps its own buffers, and vLLM only uses the share of the card set by --gpu-memory-utilization. “In total” matters for serving: the cache is shared between everyone talking to the model at once, so 100k tokens on an 80 GB card is one long document or about a dozen 8k conversations.

The practical rule: a 48 GB card such as the L40S or RTX A6000 is enough for one person, chatting or summarising a few pages. Take an 80 GB card when you paste long documents, serve a team or want FP8. Two 24 GB cards in one machine also hold a 4-bit 70B. Compare two RTX 3090s at €0.60 an hour with one RTX A6000 at €0.72; machines with several cards are listed separately in the catalogue.

Ollama or llama.cpp: one user, fewest steps

Ollama is the shortest path. On an empty PyTorch server, open the Jupyter terminal or connect over SSH:

curl -fsSL https://ollama.com/install.sh | sh
OLLAMA_CONTEXT_LENGTH=8192 OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve > /workspace/ollama.log 2>&1 &
ollama run llama3.3:70b

The default llama3.3:70b tag is the 43 GB Q4_K_M. Setting the context length on purpose matters: if the context you ask for doesn’t fit, Ollama doesn’t fail, it moves layers to the CPU. Check with ollama ps in a second terminal. The PROCESSOR column must say 100% GPU; anything like 20%/80% CPU/GPU means the speed has just dropped five- to tenfold, and the fix is a shorter context or a bigger card. With the 8-bit cache above, 16k context also fits on a 48 GB card.

llama.cpp gives you more control: exact quantisation files, cache types, how layers are split across two cards. It has to be compiled with CUDA first; the runtime comparison has the build steps. Then:

pip install -U huggingface_hub
hf download bartowski/Llama-3.3-70B-Instruct-GGUF Llama-3.3-70B-Instruct-Q4_K_M.gguf --local-dir /workspace/models

./build/bin/llama-server -m /workspace/models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  -ngl 99 -c 16384 --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 --port 8081

-ngl 99 puts every layer on the GPU. Port 8080 is taken by Jupyter on our servers, so use another one and open it through an SSH tunnel: ssh -p <port> -L 8081:localhost:8081 root@<address>, with the port and address from the server card, then go to http://localhost:8081 for llama.cpp’s built-in chat page. The SSH key guide covers the key. On two cards, llama.cpp splits the layers between them by itself.

vLLM: an API for several users

vLLM is the choice when other programs or other people send requests, because it batches them and its total throughput grows with the number of parallel requests. Our vLLM server already has vLLM installed and port 8000 published. It starts a small Qwen 7B model by itself, so stop that first, then start the 70B:

pkill -f "vllm serve"

# 48 GB card: 4-bit AWQ, short context
vllm serve casperhansen/llama-3.3-70b-instruct-awq \
  --max-model-len 8192 --gpu-memory-utilization 0.95 --max-num-seqs 8 \
  --host 0.0.0.0 --port 8000 --api-key change-me > /workspace/vllm.log 2>&1 &

# 80 GB H100: FP8, near-original quality
vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --max-model-len 16384 --gpu-memory-utilization 0.95 \
  --host 0.0.0.0 --port 8000 --api-key change-me > /workspace/vllm.log 2>&1 &

# two 24 GB cards in one machine: split every layer across both
vllm serve casperhansen/llama-3.3-70b-instruct-awq \
  --tensor-parallel-size 2 --max-model-len 8192 --gpu-memory-utilization 0.95 \
  --host 0.0.0.0 --port 8000 --api-key change-me > /workspace/vllm.log 2>&1 &

Loading takes several minutes after the download; tail -f /workspace/vllm.log shows progress. vLLM checks at start that at least one request of --max-model-len fits and refuses to start if it doesn’t, with a message naming the largest context that would fit. Lower the number or add --kv-cache-dtype fp8, which doubles the room on Ada, Hopper and Blackwell cards. The FP8 build on an H100 is the tight case: its weights take about 72 GB, so the 16k above needs the 0.95 setting, and longer contexts need the fp8 cache or a bigger card. Keep --api-key: the API address works from anywhere, and without a key anyone who finds it can use your GPU. Our vLLM server pins a vLLM version that matches the machines’ CUDA driver; if you install a newer one on an empty server, check the CUDA version shown before launch first.

How fast: tokens per second by card

For one user, every new token means reading the whole model from memory once, so speed is roughly memory bandwidth divided by model size, minus overhead. These are output tokens for a single conversation, from bandwidth arithmetic and public community reports, not our own benchmarks:

People read at about 5–8 tokens per second, so even the slowest row feels fluent in a chat. Two other numbers matter in practice. Reading the prompt is much faster than writing the answer, hundreds to a few thousand tokens per second depending on the card, so a 10-page document takes from a few seconds to half a minute before the first word. And with many parallel requests vLLM’s total output grows to several hundred or more than a thousand tokens per second on an H100. The cost per million tokens article turns those batched rates into euros.

What a first evening costs, and when a smaller model is enough

The download comes first: about 40–43 GB for a 4-bit build and about 72 GB for FP8. That is billed traffic at the machine’s per-GB rate shown before launch, plus the minutes it takes, from under ten minutes on a fast line to half an hour on a slower one. Two hours with Ollama on an L40S cost €2.12 in GPU time, two hours of FP8 on an H100 €7.80. Prices are live, and the per-minute price is locked when you launch.

If you come back often, you can keep the model in the saved folder: 43 GB costs 43 × €0.03 a month. Restoring it to the next server is traffic too, so compare that with simply downloading again; the article on files after a stop works through both.

Before you settle on 70B, try a 32B model on a 24 GB card. Qwen3 32B in 4-bit fits an RTX 4090 at €0.59 an hour, answers about twice as fast, and for a lot of everyday chat and writing the difference from a 70B is small. Run the same five real prompts through both in one session; the 70B earns its bigger card when the 32B gets your facts or your instructions wrong. The Llama model page lists the memory for each size.