toolbox
Home / AI, Data & Technology / Retrieval Augmented Generation Latency Capacity Estimator

Retrieval Augmented Generation Latency Capacity Estimator

This estimator models how many concurrent retrieval-augmented generation requests a deployment can support while staying within a target end-to-end latency. It combines retrieval time, prompt assembly, model generation time, and a utilization safety factor rather than treating model inference as the only source of delay. The result is useful when sizing an API tier, planning a load test, or deciding whether a latency target is realistic for the expected traffic.

The calculator reports estimated service time per request, theoretical requests per second per worker, and the worker count needed for the selected arrival rate. Because real RAG systems include queueing, cache effects, network variability, and changing document-hit patterns, the safety utilization input is important. Use a lower utilization target for bursty workloads or strict service-level objectives, and validate the estimate with production-like traces before committing capacity.

Inputs

ms
ms
ms
req/s
%
ms
Result
—
Concurrent workers required
Estimated service time—
Capacity per worker—
Required concurrent workers—
Capacity headroom—

1. Enter retrieval latency
Use a representative percentile from the vector database or search service, not only an ideal local benchmark.

2. Add generation and application time
Include model response time, prompt construction, routing, and other request overhead in milliseconds.

3. Set expected traffic
Enter the sustained request arrival rate you need the system to process.

4. Choose a utilization target
A value below 100% preserves queueing headroom; stricter latency goals generally need more headroom.

5. Review the capacity estimate
Compare required workers with available serving replicas and test the design under burst conditions.

Service time (seconds) = (retrieval ms + prompt ms + generation ms + overhead ms) / 1,000 Capacity per worker = target utilization / service time Required workers = expected requests per second / capacity per worker

Target utilization is converted from a percentage to a decimal. The model assumes each worker handles one active request at a time and that the entered component latencies are additive. If a worker supports parallel decoding or request batching, treat one “worker” as the effective concurrent processing slot and adjust the interpretation accordingly.

What the result means

The main result is the minimum whole number of effective concurrent workers needed at the selected utilization target.

This is a planning estimate, not a queueing simulation; burstiness and long-tail latency can require additional capacity.

Given: retrieval 120 ms, prompt assembly 25 ms, generation 850 ms, other overhead 40 ms, traffic 8 requests/second, and target utilization 70%.

Calculation: Service time = (120 + 25 + 850 + 40) / 1,000 = 1.035 seconds. Capacity per worker = 0.70 / 1.035 = 0.676 requests/second. Required workers = 8 / 0.676 = 11.83, rounded up to 12.

Result: Plan for at least 12 effective concurrent workers. At that count, estimated capacity is about 8.12 requests/second, leaving roughly 1.5% headroom above the entered load.

Why use a utilization target below 100%?

At full theoretical utilization, even small traffic bursts create a queue. A lower target leaves room for variable retrieval and generation times.

Which latency percentile should I enter?

Use a percentile that matches your service objective, often p90 or p95 for planning. An average can understate the capacity needed to protect tail latency.

Does one worker always equal one GPU?

No. A worker is an effective concurrent processing slot in this model. One GPU may host several slots, while a distributed pipeline may consume several devices per slot.

How should batching be handled?

Use measured effective service time and throughput for the batched configuration. Do not divide latency by batch size unless the serving system actually achieves that scaling.

Why can real latency exceed the estimate?

Queueing, cold caches, network contention, retries, and longer-than-expected outputs are not modeled directly. Load testing is needed to measure those effects.