The Prompt Caching Throughput Estimator estimates completed requests per second for a workload containing both cache hits and cache misses. It uses the weighted processing time per request and the number of parallel workers to show how caching may change service volume.
The calculator is useful for comparing cache strategies or checking whether a deployment can meet a traffic target. Results represent steady-state processing capacity and do not separately model queueing, burstiness, retries, or rate limits.
1. Enter hit and miss times
Use measured processing times for the same request definition.
2. Set the cache hit rate
Enter the expected share of requests served from cache.
3. Add concurrency
Specify how many workers or parallel execution slots are available.
4. Apply a utilization target
Reserve enough unused capacity for variation and operational overhead.
5. Read throughput
Review requests per second, minute, and hour.
All request types are assumed to share the same worker pool.
Throughput is the estimated steady-state number of completed requests per second.
If traffic arrives unevenly, the practical rate may be lower because queueing and tail latency grow before theoretical capacity is reached.
Given: 85% hits at 120 ms, 15% misses at 800 ms, 12 workers, and 75% utilization.
Calculation: Weighted time = 0.85 × 120 + 0.15 × 800 = 222 ms. Throughput = 12 × 0.75 × 1,000 ÷ 222 = 40.54 requests/second.
Result: About 40.54 requests per second, or roughly 145,946 per hour.
No. This estimate represents completed requests. Incoming traffic above that rate would accumulate in a queue or be rejected.
Convert seconds to milliseconds before entering them; one second equals 1,000 milliseconds.
This formula assumes a shared pool. Separate pools should be modeled independently and then combined carefully.
It prevents the estimate from assuming every worker is continuously productive with no overhead or burst margin.
Keep workers and utilization constant, then change the hit rate and processing times for each policy.