toolbox
Home / AI, Data & Technology / LLM Inference Throughput Estimator

LLM Inference Throughput Estimator

The LLM Inference Throughput Estimator projects effective cluster token throughput from GPU count, measured per-device performance, and multi-device scaling efficiency. It also converts aggregate tokens per second into approximate requests per second using the workload’s average total tokens per request, and separates input from output throughput for easier comparison with serving metrics.

Use this calculator when translating a benchmark into a deployment-level capacity estimate or when testing how scale-out efficiency changes expected output. The result is not a latency guarantee. Prefill-heavy and decode-heavy workloads stress hardware differently, and throughput can shift with context length, batch policy, sampling settings, and concurrent sequence count. Base the per-GPU value on the same workload definition used for the planned service.

Serving throughput assumptions

GPUs
tok/s
%
%
tokens
%
Result
—
effective aggregate tokens per second
Estimated requests per second—
Input token throughput—
Output token throughput—
Tokens per 24-hour day—

1. Enter GPU count
Provide the number of devices actively serving the model.

2. Add per-GPU performance
Use a measured token rate from the intended model, precision, and serving engine.

3. Apply scaling efficiency
Account for synchronization, communication, scheduler, and load-balancing losses.

4. Describe the token mix
Set the fraction of total tokens attributable to input processing.

5. Enter request size
Use average input plus generated tokens per request.

6. Review sustained output
The daily figure applies the selected availability percentage to a full 24-hour period.

Aggregate throughput = GPU count × Per-GPU throughput × Scaling efficiency Requests per second = Aggregate throughput ÷ Average tokens per request Daily tokens = Aggregate throughput × 86,400 × Availability

What the result means

The main result is estimated effective cluster throughput after the selected scaling loss.

Estimates depend on the model, serving stack, hardware, quantization, batching strategy, and workload mix. Validate the result with measurements from your own environment before committing capacity or budget.

Given

  • 8 GPUs
  • 900 tokens/s per GPU
  • 82% scaling efficiency
  • 1,450 tokens per request
  • 95% availability

Calculation

Aggregate throughput = 8 × 900 × 0.82 = 5,904 tok/s. Requests per second = 5,904 ÷ 1,450 = 4.072. Daily tokens = 5,904 × 86,400 × 0.95 = 484,496,640.

Result

5,904 effective tokens/s and about 4.072 requests/s.

The daily value assumes the same effective rate whenever the service is available.

Should input and output tokens use the same per-GPU rate?

Not necessarily. Prefill and decode performance differ. Use a blended measured rate only when the benchmark reflects the same input/output mix as production.

What is scaling efficiency?

It is the fraction of ideal linear scaling retained after adding devices. An 82% value means the cluster delivers 82% of GPU count multiplied by standalone throughput.

Can throughput exceed the request rate users observe?

Yes. Token throughput can be high while requests per second stays modest when prompts or completions are long.

How should uptime be interpreted?

It is the share of a full day during which the stated throughput is available. It can represent maintenance, deployments, idle windows, or other planned loss of serving time.

Why does this not show latency?

Throughput measures completed work per unit time, while latency measures time per request. Queueing and batching can raise throughput while worsening individual response time.