toolbox
Home / AI, Data & Technology / AI Sales Assistant Latency Capacity Estimator

AI Sales Assistant Latency Capacity Estimator

This estimator calculates expected end-to-end latency and the concurrent capacity needed for an ai sales assistant service. It combines fixed request overhead, prompt-processing time, token generation time, and optional downstream work, then relates that duration to the target request rate.

Sales teams can use it to test whether interactive drafting, account research, or live conversation assistance will feel responsive. The output is a planning estimate; queueing, network variability, cold starts, and provider throttling can increase observed latency.

Calculator inputs

ms
tokens
tok/s
tokens
tok/s
ms
req/s
Result
—
Calculated result
Prompt latency—
Generation latency—
End-to-end latency—
Required concurrency—

1. Add fixed overhead

Enter connection, routing, retrieval, and request setup time that does not scale directly with token count.

2. Describe prompt processing

Enter input tokens and the effective prompt-processing throughput.

3. Describe generation

Enter expected output tokens and the sustained decoding rate per active request.

4. Include downstream work

Add formatting, validation, storage, or other post-processing time.

5. Set arrival rate

Enter the target average requests per second for the service.

6. Read latency and concurrency

Use the total latency for service-level planning and the concurrency estimate for capacity sizing.

Prompt latency (s) = Input tokens ÷ Prompt processing rate
Generation latency (s) = Output tokens ÷ Generation rate
Total latency (s) = Fixed overhead ÷ 1,000 + Prompt latency + Generation latency + Post-processing ÷ 1,000
Required concurrency = Requests per second × Total latency

The concurrency relationship follows Little’s Law under a stable average workload. It does not include a safety margin for bursts or failures.

What the result means

The main result is estimated end-to-end completion time for one task.

Use measured workload data and current provider or hardware benchmarks when available; production results may vary.

Given: 350 ms fixed overhead, 5,600 input tokens at 4,000 tokens/s, 500 output tokens at 70 tokens/s, 180 ms post-processing, and 2.5 requests/s.

Calculation: Prompt latency = 5,600 ÷ 4,000 = 1.40 s. Generation latency = 500 ÷ 70 = 7.14 s. Total = 0.35 + 1.40 + 7.14 + 0.18 = 9.07 s. Concurrency = 2.5 × 9.07 = 22.68.

Result: Expected latency is 9.07 seconds and the average workload needs about 22.68 simultaneous processing slots before adding headroom.

Is this a p50 or p95 latency estimate?

It is a deterministic estimate based on the values entered. Use p95 component measurements when planning a p95 target, or add an explicit safety margin after calculating.

How do streaming responses affect the result?

Streaming can improve time to first token but does not necessarily reduce total generation time. This calculator estimates completion latency, not first-token latency.

Where should retrieval time be entered?

Put retrieval and orchestration time in fixed overhead unless it scales predictably with another entered variable.

Why does output length often dominate latency?

Autoregressive decoding commonly produces tokens sequentially, so generation time grows approximately with output token count divided by decoding rate.

Can required concurrency be fractional?

Yes, the result is an average. Operational capacity should be rounded up and expanded for traffic bursts, maintenance, and failed or retried requests.