toolbox
Home / AI, Data & Technology / Retrieval Augmented Generation GPU Requirements Estimator

Retrieval Augmented Generation GPU Requirements Estimator

This estimator converts a RAG workload into an approximate GPU count by combining concurrent demand, model memory, per-request working memory, and usable GPU memory. It is intended for early infrastructure planning when you need a defensible starting point for the number of accelerators required to host a model and its active requests.

The result separates memory-driven GPU demand from concurrency-driven demand and uses the larger of the two. That distinction matters because a model may fit in memory but still need more devices to serve the target number of simultaneous requests, or the opposite may be true for a large model with modest traffic. The estimate does not replace a serving benchmark: quantization, tensor parallelism, KV-cache implementation, context length, and batching policy can all materially change actual requirements.

Inputs

GB
GB
requests
GB
%
slots
Result
—
Estimated GPU count
Total estimated memory—
GPUs required by memory—
GPUs required by concurrency—
Usable memory per GPU—

1. Estimate model memory
Include weights and persistent serving overhead after the chosen precision or quantization is applied.

2. Enter per-request memory
Use a measured or estimated KV-cache and workspace amount for the typical context and output length.

3. Set peak concurrency
Enter simultaneous active requests, not requests per second.

4. Describe each GPU
Provide physical memory, usable-memory percentage, and the observed request slots one GPU can sustain.

5. Review the binding constraint
The larger of memory demand and concurrency demand determines the displayed GPU count.

Total memory = model memory + (memory per active request × concurrent requests) Usable memory per GPU = GPU memory × usable memory percentage GPUs by memory = total memory / usable memory per GPU GPUs by concurrency = concurrent requests / request slots per GPU Estimated GPUs = ceiling(max(GPUs by memory, GPUs by concurrency))

The method assumes the model can be partitioned across the resulting devices and that entered memory figures already reflect the serving stack. It does not model interconnect bandwidth or compute saturation.

What the result means

The result is the minimum whole GPU count implied by the larger of memory capacity and concurrent request-slot demand.

A configuration that fits in memory may still miss throughput or latency goals, so benchmark the selected hardware and serving engine.

Given: 42 GB of model and runtime memory, 1.8 GB per active request, 24 concurrent requests, 80 GB per GPU, 85% usable memory, and 8 request slots per GPU.

Calculation: Total memory = 42 + (1.8 × 24) = 85.2 GB. Usable memory per GPU = 80 × 0.85 = 68 GB. GPUs by memory = 85.2 / 68 = 1.25, rounded up to 2. GPUs by concurrency = 24 / 8 = 3.

Result: The binding requirement is concurrency, so the estimate is 3 GPUs. This configuration provides 204 GB of usable aggregate memory and 24 modeled request slots.

What should be included in model memory?

Include model weights plus persistent runtime allocations such as kernels, buffers, and serving-engine overhead. Use measurements from the intended precision and framework when available.

How do I estimate memory per request?

KV-cache size is usually the main variable component and depends on model architecture, context length, batch behavior, and precision. Measure it with representative prompts whenever possible.

Does the calculator account for tensor parallelism?

It assumes the total memory can be distributed across the estimated devices. It does not estimate communication cost or whether a particular parallelism layout is efficient.

Can CPU offload reduce the GPU count?

It may reduce GPU memory pressure, but often adds transfer latency and can lower throughput. Enter the post-offload GPU memory footprint only after testing the intended setup.

Why is concurrency sometimes the binding constraint?

A model can fit on one GPU while the service still needs more GPUs to process many simultaneous requests at the target latency. Request-slot demand captures that operational limit.