The Prompt Caching GPU Requirements Estimator approximates how many GPUs are needed to serve a cached-prompt workload at a target request rate. It combines cache-hit and cache-miss GPU time, adds utilization headroom, and rounds the requirement up to a whole device.
This is useful for early infrastructure sizing and scenario comparison. It does not replace load testing because memory limits, model parallelism, batch efficiency, and cache storage architecture can change the practical device count.
1. Set traffic demand
Enter the sustained requests per second to be served.
2. Describe cache performance
Provide the hit rate plus GPU processing time for hits and misses.
3. Reserve operating headroom
Set the fraction of theoretical GPU capacity considered safely usable.
4. Use the rounded result
Provision at least the displayed whole-GPU count, then verify with a load test.
The result is rounded up because fractional devices cannot normally be provisioned independently.
The output is the minimum whole-device estimate at the entered average load and utilization target.
Memory capacity, model replicas, failure tolerance, and multi-GPU sharding may require more devices than the compute-only estimate.
Given: 40 requests/second, 80% cache hits, 45 GPU-ms per hit, 420 GPU-ms per miss, and 70% usable utilization.
Calculation: Average GPU time = 0.80 × 45 + 0.20 × 420 = 120 GPU-ms. Raw demand = 40 × 120 ÷ 1,000 = 4.8 GPUs. Adjusted demand = 4.8 ÷ 0.70 = 6.86.
Result: Round up to 7 GPUs.
A capacity requirement of 6.1 GPUs still needs seven whole devices unless the platform supports fractional allocation.
Yes, if a hit is served without GPU inference. Enter zero when that reflects the actual architecture.
No. It estimates compute demand only; model weights, KV cache, batching, and replicas must also fit in memory.
Use the rate the system must sustain. For production sizing, a peak or high-percentile rate is usually more useful than a daily average.
A higher hit rate lowers the weighted GPU time when hits consume less GPU compute than misses.