toolbox
Home / AI, Data & Technology / LLM Inference GPU Requirements Estimator

LLM Inference GPU Requirements Estimator

The LLM Inference GPU Requirements Estimator calculates a starting GPU count from two independent constraints: aggregate token throughput and model-serving memory. It scales measured per-GPU throughput by a sustainable utilization target, compares that requirement with the number of devices needed to hold model and runtime memory, and then adds explicit redundancy capacity.

The result is intended for early cluster sizing and procurement comparisons. Actual GPU needs depend on tensor or pipeline parallelism, quantization, KV-cache growth, context length, batch composition, interconnect bandwidth, and the serving engine. Benchmark the exact model, precision, and workload on candidate hardware; a published peak specification is not a substitute for measured tokens per second in your stack.

Throughput and memory assumptions

tok/s
tok/s
%
GPUs
GB
GB
Result
—
recommended GPU count
GPUs required by throughput—
GPUs required by memory—
Effective throughput per GPU—
Total usable GPU memory—

1. Set the service target
Enter the aggregate token throughput the deployment must sustain.

2. Use measured GPU throughput
Provide tokens per second achieved by one GPU under a representative model, precision, batch, and context mix.

3. Choose sustainable utilization
Reduce benchmark throughput to a level that leaves operating margin.

4. Add redundancy
Enter spare devices required for maintenance or failure tolerance.

5. Check memory fit
Provide usable memory per GPU and the combined model-plus-runtime memory footprint.

6. Review the binding constraint
Compare throughput-driven and memory-driven counts; the larger requirement determines the base cluster size.

Effective throughput per GPU = Measured GPU throughput × Utilization Throughput GPUs = ceil(Target throughput ÷ Effective throughput per GPU) Memory GPUs = ceil(Model and runtime memory ÷ Usable memory per GPU) Recommended GPUs = max(Throughput GPUs, Memory GPUs) + Redundancy

What the result means

The main result is a whole-device estimate that satisfies both entered constraints before topology-specific adjustments.

Estimates depend on the model, serving stack, hardware, quantization, batching strategy, and workload mix. Validate the result with measurements from your own environment before committing capacity or budget.

Given

  • 6,000 target tokens/s
  • 850 measured tokens/s per GPU
  • 75% sustainable utilization
  • 1 redundant GPU
  • 72 GB usable memory per GPU
  • 132 GB model and runtime memory

Calculation

Effective throughput = 850 × 0.75 = 637.5 tok/s. Throughput GPUs = ceil(6,000 ÷ 637.5) = 10. Memory GPUs = ceil(132 ÷ 72) = 2. Recommended = max(10, 2) + 1 = 11.

Result

11 GPUs.

Throughput is the binding constraint in this scenario; the memory minimum alone would be much smaller.

Does the memory calculation include KV cache?

Only when you include it in the model and runtime memory input. Estimate KV-cache memory for the expected concurrent sequences and context lengths rather than entering model weights alone.

Why add redundancy after taking the maximum?

Redundancy represents extra devices beyond the capacity needed to meet the stronger base constraint. It should reflect your maintenance and failure policy.

Can I compare different GPU types with this page?

Yes, run one estimate per device type using measured per-GPU throughput and usable memory for each candidate. Also compare power, cost, and interconnect requirements.

How does model parallelism affect the count?

Some models require a minimum parallel group size or specific topology. Round the recommendation up to a valid group multiple when your serving configuration requires it.

Is tokens per second enough for sizing?

No. Confirm time to first token, inter-token latency, request-level throughput, memory, and tail behavior. Two systems with similar aggregate throughput may deliver very different user experience.