12 November 2026 · VRAMfarm team

H200 vs H100: when 141 GB is worth paying for, and when 80 GB is enough

Llama 3.3 70B in fp8 loads on an H100 with only 4–8 GB to spare. That leaves room for roughly 12–25 thousand tokens of 16-bit KV cache, shared by every conversation the server holds. On an H200 the same model leaves about 60 GB, enough for the full 128k context or twenty parallel chats of 8k each. That is the case for the H200 in one example: it has the same compute as the H100, with 141 GB of memory instead of 80 and 4.8 TB/s of bandwidth instead of 3.35. It rents from €5.43 an hour, the H100 from €3.90. Whether the extra memory is worth it depends on whether your job runs out of memory on the H100, and that is easy to check.

Same chip, more and faster memory

Both cards run exactly the same software, so moving a project from one to the other needs no changes. The catalogue groups a few versions under each name: the H100 can be SXM, PCIe (about 2 TB/s, lower clocks) or the 94 GB H100 NVL; the H200 can be SXM or the PCIe-based H200 NVL. The exact name of each machine is shown before you launch.

Two consequences follow from the table. Anything compute-bound runs at the same speed on both: training a model that fits on both, image and video generation, prompt processing. Anything memory-bound runs up to 1.4 times faster on the H200, because 4.8 ÷ 3.35 ≈ 1.43. That is mainly text generation, where each new token reads all the model’s weights.

When 80 GB runs out

Memory is where the two cards really differ. The estimates below use the formula from the VRAM guide: weights, plus KV cache, plus a few gigabytes for the runtime. For Llama 3.3 70B, the cache takes about 0.33 MB per token in 16-bit and half that in fp8.

The pattern: the H200 matters for 70B models in fp8, long context on any large model, and the big mixture-of-experts models. A mixture-of-experts model such as Qwen3 235B-A22B only computes with about 22B parameters per token, so it generates text quickly, but all 235B parameters have to sit in memory. On one H200, a lean 4-bit build fits with a moderate context; heavier 4-bit formats do not, and then a 192 GB B200 or two cards are the answer.

Why fp8 instead of 4-bit for a 70B model? fp8 is very close to the original model in quality and runs fast on Hopper’s fp8 tensor cores. 4-bit is smaller and fine for chat, but loses a little more on code, maths and long reasoning. If 4-bit is good enough for your use, the H100 has room for it and is the cheaper card (see the next section).

For a 70B model at normal context, you may not need either card: a 4-bit 70B runs on a 48 GB card with 8–12k tokens of context, at a fraction of the price.

Speed: what the extra bandwidth buys

For one stream of text, the H200 is about 1.3–1.4 times faster: a 70B model in fp8 generates roughly 25–38 tokens per second on an H100 and 35–55 on an H200, depending on the runtime and context length.

For a server with many users, the gain can be much larger, but for a different reason. Batching lets the card generate tokens for many conversations with one read of the weights, and the batch size is limited by how much KV cache fits. On an H100 with a 70B fp8 model, the cache only holds a few conversations at a time; on an H200 it holds many times more. NVIDIA quotes up to about 1.9 times the H100’s throughput on 70B inference; how much of that you see depends on how far your batch was limited by memory on the H100.

For training and image generation, expect the same speed within a few per cent, unless the extra memory changes your setup. A full fine-tune of a 7–8B model with standard AdamW takes about 16 bytes per parameter, 112–128 GB before activations. On an H200 that can fit on one card with gradient checkpointing and short sequences; on an H100 it needs an 8-bit optimiser or two cards. Removing sharding or offloading saves time, but the compute per step stays the same.

Price per useful token

What counts is the cost per finished job, not per hour. The figures are output tokens in a long vLLM batch, with ranges because they depend on prompt and answer lengths, parallel requests and the vLLM version. The same assumptions are used in the cost per million tokens article.

The cost columns use live prices, so they show today’s answer. With the prices of November 2026 the result was:

• 70B in fp8: the H200 is clearly cheaper per token, because the H100 is starved of cache memory and cannot batch.
• 70B in 4-bit: the H100 is usually cheaper, because it has enough cache room and the H200’s bandwidth gain is smaller than its price difference.
• Small models: the H100, because both cards are mostly compute-bound there.

A short job still pays the 15-minute minimum per launch, so these rates matter for batches of tens of millions of tokens, not for a quick test.

One H200 or two H100s?

Two H100s give 160 GB and twice the compute for €7.80 an hour, against €5.43 for one H200. On paper the pair wins. In practice it depends on the machine and the software:

• On an SXM machine with NVLink, a 70B model split over two H100s with vLLM’s --tensor-parallel-size 2 gives high throughput and plenty of cache room. If the pair costs about the same as one H200, it is usually the faster option for serving.
• On PCIe H100s without a bridge, splitting a model over two cards sends data between them on every layer, and a good part of the gain is lost. nvidia-smi topo -m after launch shows how the cards are connected; multi-GPU training basics explains how to read it.
• One H200 is simpler. No parallel settings, no inter-card traffic, and it works with tools that only use one GPU. For a mixture-of-experts model that fits on one card, keeping it on one card avoids sending tokens between cards to reach their experts.

Which one to rent

• 70B in fp8, long documents, many parallel users, or a 235B MoE model in a lean 4-bit build: the H200.
• 70B in 4-bit, 32B models, fine-tuning a model that fits in 80 GB, image or video work: the H100, or a cheaper card. For images, a consumer card is much cheaper per image (see H100 vs A100 vs RTX 4090).
• Not sure: start the H100 with your real settings. If vLLM refuses to start at the context you need, or the “maximum concurrency” it prints at startup is only a handful of requests, move to the H200. With per-minute billing, the test costs the 15-minute minimum, about €0.98.

The H200 and H100 pages show the machines available right now, with the exact card version, country, driver and CUDA version of each.