toolbox
Home / AI, Data & Technology / Retrieval Augmented Generation Throughput Estimator

Retrieval Augmented Generation Throughput Estimator

This estimator calculates expected RAG request throughput from the number of serving workers, average end-to-end service time, and a practical utilization target. It is designed for capacity planning when retrieval and generation operate as one user-visible transaction and you need to translate measured latency into a sustainable requests-per-second figure.

The output includes total request throughput and an estimated daily task volume. It also shows how much of each request is spent in retrieval versus generation and other processing, which helps identify where optimization work may have the greatest effect. The calculation assumes a stable workload and independent effective workers. Real systems may achieve different results because of dynamic batching, shared vector-store bottlenecks, rate limits, queueing, and a wide distribution of prompt and output lengths.

Inputs

workers
ms
ms
ms
%
hours
Result
—
Sustainable requests per second
Service time per request—
Estimated tasks per day—
Retrieval share of latency—
Throughput per worker—

1. Count effective workers
Use processing slots that can independently advance requests, rather than physical servers unless they map one-to-one.

2. Enter measured component times
Add representative retrieval, model generation, and other application latency in milliseconds.

3. Select sustainable utilization
Leave headroom for variability; the value should reflect a level your service can maintain without unstable queues.

4. Set daily operating time
Use 24 hours for continuous service or a smaller active window for business-hour demand.

5. Inspect throughput and mix
Review requests per second, daily volume, and the retrieval share before changing infrastructure or code.

Service time (seconds) = (retrieval ms + generation ms + other ms) / 1,000 Throughput per worker = utilization / service time Total throughput = workers × throughput per worker Daily tasks = total throughput × operating hours × 3,600

Utilization is expressed as a decimal in the formula. Each effective worker is assumed to process one active request at a time; use measured effective slots if your serving engine multiplexes or batches work.

What the result means

The main result estimates the sustained RAG requests per second supported by the entered worker pool.

It represents steady-state capacity and does not guarantee a particular tail-latency percentile during bursts.

Given: 16 workers, 90 ms retrieval, 760 ms generation, 50 ms other processing, 75% utilization, and 24 operating hours.

Calculation: Service time = (90 + 760 + 50) / 1,000 = 0.9 seconds. Throughput per worker = 0.75 / 0.9 = 0.833 requests/second. Total throughput = 16 × 0.833 = 13.33 requests/second. Daily tasks = 13.33 × 24 × 3,600 = 1,152,000.

Result: Estimated sustainable throughput is 13.33 requests per second, or about 1.15 million tasks per full day. Retrieval accounts for 10% of modeled service time.

Should I use average or percentile latency?

Use a representative service time for sustained capacity planning, then separately validate tail latency. A high percentile produces a more conservative throughput estimate.

How does dynamic batching affect the result?

Batching can increase effective throughput without reducing request latency proportionally. Enter an effective worker count or measured service time that reflects the batched setup.

What if retrieval and generation overlap?

This formula adds component times. If your architecture overlaps them, enter the measured end-to-end service time by assigning the non-overlapped portion across the fields.

Can the daily task estimate be treated as a quota?

No. It is a steady-state volume estimate based on constant utilization during the entered operating hours. Maintenance, bursts, and traffic troughs change actual volume.

How can I identify the best optimization target?

Compare each component’s share of service time. Reducing the largest component usually offers the greatest first-order throughput improvement, provided no new bottleneck appears.