LLM VRAM Calculator

LLM VRAM is the GPU memory a large language model occupies while it runs: the bytes that store its weights plus a key-value cache that grows with context length and batch size. This LLM VRAM calculator estimates that total in GB and GiB, splits it into weights, KV cache and overhead, and shows which common card sizes it fits on.

How much VRAM do you need to run an LLM?

Enough to hold two things at once. The first is the model itself: every parameter is a number, and each number takes a fixed count of bytes that depends on the precision it is stored in. The second is working memory for the text being processed, most of which is the key-value (KV) cache that lets the model generate one token at a time without recomputing the whole prompt.

The weights are a fixed cost. You pay it once when the model loads, whether the prompt is ten tokens or ten thousand. The cache is a variable cost that scales with how much text is in play. That is why a model can load without trouble and still run out of memory half way through a long document.

What is the formula for LLM memory?

This GPU memory calculator for LLM inference uses a deliberately simple model:

• \(P\) is the number of parameters and \(b_w\) the bytes per parameter: 4 for float32, 2 for float16 or bfloat16, 1 for 8-bit and 0.5 for 4-bit.
• \(L\) is the number of transformer layers, \(T\) the context length in tokens and \(B\) the number of sequences held in memory at the same time.
• \(d\) is the hidden size, \(h\) the number of attention heads and \(h_{kv}\) the number of key-value heads. Each head works on \(d / h\) numbers, so one token stores \(d \, h_{kv} / h\) numbers per key and the same again per value. The leading 2 covers both.
• \(b_{kv}\) is the bytes per cached number (2 by default) and \(o\) is the overhead as a fraction, 0.10 for 10%.

The first term is the weights and the second is the cache. If you only know the architecture, the neural network parameter calculator gives you \(P\) from the layer sizes.

Worked example: a 7B-class model at float16

The calculator loads with 7 billion parameters at float16, 32 layers, hidden size 4096, 32 attention heads, 32 key-value heads, a 4,096-token context, batch size 1, a 2-byte cache and 10% overhead.

• Weights: 7,000,000,000 × 2 = 14,000,000,000 bytes, which is 14 GB or 13.04 GiB.
• KV cache: 2 × 32 × 4096 × 1 × 4096 × (32 / 32) × 2 = 2,147,483,648 bytes, which is exactly 2 GiB or 2.15 GB.
• Total: (14,000,000,000 + 2,147,483,648) × 1.10 = 17,762,232,012.8 bytes, which is 17.76 GB or 16.54 GiB. The overhead share is 1.61 GB.

The estimate does not fit a 16 GB card, even though the weights alone would. It fits a 24 GB card with about 7.46 GiB to spare. That gap between “the weights fit” and “the workload fits” is the reason to count more than parameters times bytes.

How does quantization change the memory?

Quantization stores each weight in fewer bits. Memory for the weights falls in direct proportion, while the cache and the overhead rate stay as they were. The table uses the same 7B-class shape and the default 2 GiB cache with 10% overhead.

The trade-off is general: lower precision saves memory and can cost some output quality, and how much depends on the model and the quantization method. Treat the 8-bit and 4-bit rows as lower bounds, because quantized formats often keep a few layers at higher precision and carry some extra metadata. The Hugging Face Transformers quantization documentation describes the options available when loading a model.

Why does the KV cache decide whether a long prompt fits?

Attention in each layer is a set of matrix products between the current token and the keys and values of every earlier token. Rather than recompute those for each new token, the runtime stores them. The store holds one key and one value per token, per layer, per sequence, so it grows linearly with context length and with batch size.

These rows use the 7B-class shape with a 2-byte cache. At 32,768 tokens the cache alone is 17.18 GB, more than the 14 GB of float16 weights. Serving four sequences at once multiplies the cache by four and leaves the weights unchanged.

Grouped-query attention cuts this down by sharing key-value heads across several query heads. With 8 key-value heads instead of 32, the 4,096-token cache falls from 2 GiB to 0.5 GiB. Use the context your runtime actually allocates: some reserve the full window up front, others grow the cache as tokens arrive.

What does the estimate leave out?

Three things, which is why every figure on this page is labelled an estimate.

• Activation memory during generation. Each forward pass creates temporary tensors, including the scores over the vocabulary that feed the final softmax step. They are short-lived but they need room at the peak.
• Framework overhead that varies by runtime. Libraries reserve memory pools, load kernels and keep buffers. The flat percentage here is a placeholder for that; measure your own stack and adjust it.
• Training. Training keeps activations for every layer, a gradient for every trainable parameter and optimizer state on top of the weights, so it needs far more memory than inference. When you plan a training run, size the memory separately and use the training steps calculator for the schedule.

Where is a VRAM estimate used?

The same arithmetic answers several practical questions. Before downloading a model, it tells you which precision your card can hold. When configuring an inference server, it shows how many concurrent sequences fit beside the weights and what maximum context you can promise. When comparing hosting with a paid endpoint, it gives the hardware side of the comparison, and the LLM token cost calculator gives the other side. More tools for model sizing are on the deep learning calculators page.

Common mistakes

• Counting only the weights. Parameters times bytes ignores the cache, which can exceed the weights at long contexts.
• Mixing GB and GiB. 14 GB is 13.04 GiB. A file size in GB and a memory readout in GiB are not the same number.
• Forgetting concurrency. Every sequence in a batch needs its own cache.
• Assuming a quantized model has a quantized cache. Cache precision is set separately and is often still a 16-bit float.
• Using the number of attention heads when the model has fewer key-value heads. That overstates the cache by the ratio between them.
• Treating the whole card as free. The display and other processes hold some memory before the model loads.

How do you compute model memory in PyTorch?

For a model you have already loaded, PyTorch can report the exact weight memory: multiply each parameter tensor’s element count by its element size and add them up. The second half of the snippet reproduces the worked example.

import torch

def weight_bytes(model):
    return sum(p.numel() * p.element_size() for p in model.parameters())

# One 4096 x 4096 linear layer with a bias, stored as float16
layer = torch.nn.Linear(4096, 4096).half()
print(weight_bytes(layer))        # 33562624  = (4096 * 4096 + 4096) * 2

# The worked example: 7 billion parameters at float16
weights = 7e9 * 2
kv_cache = 2 * 32 * 4096 * 1 * 4096 * (32 / 32) * 2
total = (weights + kv_cache) * 1.10

print(weights / 1e9)              # 14.0
print(kv_cache / 2**30)           # 2.0
print(round(total / 1e9, 6))      # 17.762232
print(round(total / 2**30, 6))    # 16.542368

torch.Tensor.element_size returns the bytes used by one element, so the sum is correct for any mix of dtypes. In Keras the equivalent starting point is model.count_params(), multiplied by the bytes per weight of the dtype in use.

Frequently asked questions

How much VRAM do I need to run a 7B model?

At float16 a model with 7 billion parameters needs 14 GB for its weights alone. With a 4,096-token context, one sequence and 10% overhead, this estimator gives 17.76 GB (16.54 GiB). At 4-bit precision the weights shrink to 3.5 GB and the same estimate drops to 6.21 GB.

Does context length affect VRAM?

Yes. The key-value cache stores a key and a value vector for every token in every layer, so it grows in direct proportion to context length and to the number of sequences processed at once. Doubling the context doubles the cache, while the weights stay the same size.

What is the difference between GB and GiB?

A gigabyte (GB) is 1,000,000,000 bytes and a gibibyte (GiB) is 1,073,741,824 bytes, about 7.4% more. Model files are usually quoted in GB, while GPU tools often report memory in binary units, so 14 GB of weights is 13.04 GiB.

Can I use this estimate for training or fine-tuning?

No. It covers inference only. Training also keeps activations for every layer, a gradient for every trainable parameter and optimizer state, which together can need several times the memory of the weights. Treat the result here as a floor for training, not a forecast.

Related calculators

Start from the architecture with the parameter count calculator, then compare self-hosting against a metered endpoint with the token cost estimator.

More tools that pair with this one:

• Steps and epochs for a training run
• Cross-entropy loss for a prediction
• All deep learning calculators