The Prompt Caching Latency Capacity Estimator calculates expected response latency and the number of concurrent requests a cached prompt workflow can support. It separates cache-hit and cache-miss behavior so teams can see how hit rate changes both average processing time and service capacity.
Use it when sizing an API tier, planning an inference gateway, or checking whether a cache strategy can meet a latency target. The outputs are planning estimates: actual performance also depends on batching, model choice, network delay, queueing, and provider-specific cache behavior.
1. Enter cache behavior
Provide the expected hit rate and measured or benchmarked hit and miss latencies.
2. Set parallel capacity
Enter the number of requests that can be processed simultaneously.
3. Choose utilization
Use a utilization target below 100% to leave room for traffic variation.
4. Review latency and throughput
Compare the weighted latency with the estimated sustainable requests per second.
Hit rate is entered as a decimal in the calculation. Latencies are in milliseconds, and capacity is reported in requests per second.
The main result is the expected latency across a mix of cache hits and misses. Capacity estimates how much traffic the specified worker pool can sustain at the selected utilization.
Queueing delay and network overhead are not modeled separately, so production measurements should be used when available.
Given: 80% hit rate, 180 ms hit latency, 900 ms miss latency, 16 workers, and 75% utilization.
Calculation: Average latency = 0.80 × 180 + 0.20 × 900 = 324 ms. Capacity = 16 × 0.75 × 1,000 ÷ 324 = 37.04 requests/second.
Result: The workflow is expected to average 324 ms and support about 37.04 requests per second.
Cache misses are included in the weighted average and can add substantial processing time even when they are relatively uncommon.
Usually not for capacity planning. A lower target leaves headroom for bursts, retries, and variation in request complexity.
Only if the latency values you enter already include it. The formula treats each latency input as an end-to-end value.
At 0%, the result equals miss latency. At 100%, it equals hit latency.
Use percentile latency measurements and run separate scenarios for normal and peak traffic.