LLM inference speed estimator
Tokens per second for a model size, quantisation and GPU — the roofline numbers vLLM and llama.cpp are bounded by.
Result
The result will appear here.Before renting a GPU or buying one, you want a number: how many tokens per second will this model produce on this card? The honest answer comes from two limits. Generating a token reads every weight once, so a single stream is bound by memory bandwidth divided by the model's size in bytes; batching more users multiplies that until the card runs out of compute, at roughly two floating-point operations per parameter per token. This estimator applies both limits, with realistic efficiency factors, across two dozen common GPUs from a T4 to a B200 and Apple's unified-memory chips, and tells you which limit you hit.
How it works
- Enter the parameter count in billions and the quantisation; the weight size follows from llama.cpp's bits-per-weight for each GGUF format, with F16 for unquantised serving.
- Decode speed is the smaller of the bandwidth bound (× batch size) and the compute bound; prefill speed and time-to-first-token come from the compute bound alone.
- Every GPU in the table is estimated at once, so you can read off which card gives the per-user speed you need and whether the weights fit at all.
Where your data goes
Nowhere. This tool runs entirely in your browser: the text you paste is processed by the page and is never transmitted to a server or written to a log.
This tool is free and needs no account. Its results exist only in your open page and are not saved anywhere.
What it costs
This tool is free, with no sign-in and no points.
Common questions
- How close are these to real benchmarks?
- Usually within 20–30% for a well-tuned engine at the same batch size. The estimator assumes 70% of peak bandwidth and 50% of peak fp16 compute, which is what vLLM, TensorRT-LLM and llama.cpp reach on large matrices; a small model, a long context or a CPU offload will land lower. Treat the numbers as planning figures.
- Why does batching help so much, and when does it stop?
- At batch 1 the GPU reads 140 GB of weights to produce one token and its arithmetic units sit idle. Serving 64 users reads the same weights once per step and does 64 times the maths, so throughput rises almost linearly until compute saturates — the table marks that crossover as compute-bound. Per-user speed, however, falls as the batch grows.
- Does 'fits' include the KV cache?
- No, it compares weights alone with 90% of memory, because the cache depends on context length and concurrency, which the parameter count alone cannot tell you. Use the model size calculator for the per-token KV figure and the VRAM calculator for a full budget on a specific model.
The open-source behind it
This tool is a self-contained implementation. vllm-project/vllm (Apache-2.0) does the same job as a library — if you need this behaviour inside your own program, start there rather than calling a web page.
vllm-project/vllmAlso known as
- llm tokens per second calculator
- inference speed estimate gpu
- how fast will a 70b model run
- rtx 4090 llm speed
- h100 tokens per second
- vllm throughput estimate