Why would a 3-GPU estimate become a 4-GPU deployment?

Tensor parallelism in common serving engines splits attention heads evenly across GPUs, so the GPU count must divide the model's attention-head count. Models with 64 attention heads split across 2, 4 or 8 GPUs, not 3, so a 168 GB FP16 estimate is usually served on four 80 GB GPUs, which also leaves room for KV cache.

Model Deployment Company · Field notes / 003 · Sizing

How much GPU memory does a 70B model need?

Q: How much GPU memory does a 70B model need?

At FP16 or BF16, the weights of a 70-billion-parameter model take 140 GB (70 billion × 2 bytes). Adding about 20% for runtime overhead gives a planning estimate of about 168 GB, or three 80 GB-class accelerators. KV cache for long contexts and concurrent requests comes on top of that.

Q: Can a 70B model run on a single 80 GB GPU?

Not at FP16. With 4-bit weight quantization the weights take about 35 GB (roughly 42 GB planned), which fits on one 80 GB-class accelerator with room left for KV cache. Validate output quality on your own task before committing to a quantized model.

Q: How much memory does a 70B model need at INT8?

About 70 GB for the weights (70 billion × 1 byte), or roughly 84 GB with 20% runtime overhead, which usually means two 80 GB-class accelerators.

Published September 24, 2026 · Planning estimates, not a capacity commitment

At FP16/BF16 the weights of a 70B model take 140 GB. Add about 20% for the runtime and you should plan for ~168 GB: three 80 GB-class accelerators, which in practice usually means four. At INT8 plan for ~84 GB (two accelerators); at INT4, ~42 GB (one). KV cache for long contexts and concurrent users comes on top, so size it from your real workload.

The math

# weights = parameters × bytes per parameter 70,000,000,000 × 2 bytes (FP16/BF16) = 140 GB # planning estimate = weights × 1.2 (runtime overhead) 140 GB × 1.2 = 168 GB # accelerators = ceil(estimate ÷ 80 GB) ceil(168 ÷ 80) = 3 → usually deployed on 4 (see tensor parallelism)

This is the same rule the sizing simulator on our homepage uses. GB here are decimal gigabytes; an "80 GB" accelerator's usable memory is a little lower after the driver and runtime reserve their share.

70B by precision

Precision	Bytes / param	Weights	Planned (×1.2)	80 GB GPUs
FP16 / BF16	2	140 GB	168 GB	3
INT8	1	70 GB	84 GB	2
INT4	0.5	35 GB	42 GB	1

Quantization cuts memory but can change output quality. Evaluate the quantized model on your own task before you commit hardware to it.

Other model sizes at FP16

Model size	Weights	Planned (×1.2)	80 GB GPUs
8B	16 GB	20 GB	1
32B	64 GB	77 GB	1
70B	140 GB	168 GB	3
120B	240 GB	288 GB	4
405B	810 GB	972 GB	13

A 32B model at 77 GB technically fits one 80 GB card, but leaves almost nothing for KV cache. Treat anything within ~10% of a card's capacity as the next size up.

KV cache: the part the rule of thumb leaves out

Every token in every active request keeps a key and a value vector per layer. That memory grows with context length and with the number of requests served at once, and at high concurrency it can rival the weights.

# KV bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes # example: Llama-3-70B-style config, 80 layers, 8 KV heads, head dim 128, FP16 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.33 MB per token 8,192-token request ≈ 2.7 GB 32 concurrent 8K requests ≈ 86 GB

Illustrative. Read num_hidden_layers, num_key_value_heads and the head dimension from your model's config.json. Models without grouped-query attention store far more per token.

Why three GPUs usually becomes four

A 70B model at FP16 doesn't fit on one card, so serving engines split it with tensor parallelism. They divide attention heads evenly across GPUs, so the GPU count has to divide the head count. A model with 64 attention heads splits across 2, 4 or 8 GPUs, not 3. The 168 GB estimate therefore usually lands on four 80 GB accelerators, and the fourth card's headroom goes to KV cache, which is often exactly what the workload needs.

What to bring to a sizing conversation

The exact model and version, or the task if the model is undecided.
Precision you can accept, and how you'll judge quality.
Typical and maximum context length, and expected concurrent requests.
The accelerators you already have, or the cloud instance types you can use.

The readiness checklist covers the other decisions, and the acceptance worksheet helps you agree on what "working" means before deployment.

Run the sizing simulator Start a deployment brief

Questions engineers ask

Can a 70B model run on a single 80 GB GPU?

Not at FP16. At INT4 the weights take about 35 GB (~42 GB planned), which fits on one 80 GB-class card with room for KV cache. Check quality on your task first.

How much memory does a 70B model need at INT8?

About 70 GB of weights, or ~84 GB planned: usually two 80 GB-class accelerators.

Is 20% overhead always enough?

No. It's a planning rule for the weights plus runtime buffers. Long contexts and high concurrency add KV cache on top, which you should size separately using the formula above.

Does this apply to embedding models?

The weight math does, and embedding models are usually far smaller: a 0.3B model needs well under 1 GB of weights at FP16. Their throughput limits usually come from batch size and sequence length rather than memory.