toolbox
Home / AI, Data & Technology / LLM Inference Latency Capacity Estimator

LLM Inference Latency Capacity Estimator

The LLM Inference Latency Capacity Estimator converts a peak arrival rate and measured end-to-end latency into an approximate concurrency requirement. It applies Little’s Law to estimate requests in flight, then increases the worker count so the serving pool can operate below a chosen utilization ceiling and absorb traffic headroom.

This estimate helps teams size queues, replicas, or concurrent execution slots before a load test. It does not predict tail latency by itself: token lengths, batching, prefill contention, network time, and scheduler behavior can widen the latency distribution even when average capacity appears sufficient. Use the output as an initial concurrency target and verify p95 or p99 latency under a representative load profile.

Latency and concurrency assumptions

req/s
s
%
%
workers
s
Result
—
required concurrent workers
Raw concurrency—
Headroom-adjusted request rate—
Capacity with available workers—
Latency objective check—

1. Measure peak arrivals
Enter the highest sustained request rate the service should handle, not a brief one-second spike unless that spike must be served without queueing.

2. Enter observed latency
Use end-to-end time from request acceptance to completed response for the target model and payload mix.

3. Set utilization
Choose a worker utilization target below 100% to preserve scheduling and recovery margin.

4. Add traffic headroom
Increase the arrival rate for forecast error, bursts, or growth.

5. Enter current workers
Provide the number of concurrent execution slots already available.

6. Compare capacity and objective
Review required workers, estimated request capacity, and whether average latency meets the stated objective.

Adjusted arrival rate = Peak rate × (1 + Headroom) Raw concurrency = Adjusted arrival rate × Average latency Required workers = ceil(Raw concurrency ÷ Target utilization)

Arrival rate is requests per second, latency is seconds, and utilization is a decimal. The method assumes a stable flow and treats one worker as one concurrent request slot.

What the result means

The main result is the minimum whole-number concurrency target after headroom and utilization margin.

Estimates depend on the model, serving stack, hardware, quantization, batching strategy, and workload mix. Validate the result with measurements from your own environment before committing capacity or budget.

Given

  • 12 requests per second
  • 2.4 seconds average latency
  • 70% target utilization
  • 25% headroom

Calculation

Adjusted rate = 12 × 1.25 = 15 req/s. Raw concurrency = 15 × 2.4 = 36. Required workers = ceil(36 ÷ 0.70) = 52.

Result

52 concurrent workers.

A 60-worker pool has nominal capacity above the adjusted arrival rate, but tail-latency testing is still required.

Why use average latency if the SLO is based on p95?

Average latency is useful for estimating mean concurrency, while p95 or p99 determines user-facing reliability. Capacity should be validated with the full latency distribution.

What counts as a worker?

A worker is any independent concurrent execution slot in your architecture, such as a replica slot, scheduler slot, or process that can actively serve one request under the model used here.

Can continuous batching make this estimate inaccurate?

Yes. Continuous batching can allow several sequences to share GPU work, so a worker may not map cleanly to one request. Treat the result as logical concurrency, then translate it through measured serving efficiency.

Should queue time be included in latency?

Include it when estimating user-visible concurrency and SLO performance. Excluding queue time can understate the number of requests simultaneously present in the system.

What if traffic is highly bursty?

Use a short-window peak rate and larger headroom, or run a queueing simulation. Little’s Law alone does not describe burst shape or waiting-time variance.