toolbox
Home / AI, Data & Technology / AI Sales Assistant Throughput Estimator

AI Sales Assistant Throughput Estimator

This estimator calculates sustainable task throughput for an ai sales assistant deployment. It starts with the number of GPUs and measured token-processing capacity, applies a utilization target, and divides usable capacity by the total tokens consumed by each task.

Use it to compare expected sales-assistant demand with serving capacity across drafting, enrichment, and live-agent scenarios. The result represents steady-state capacity; traffic bursts, queue limits, and latency objectives may require additional headroom.

Calculator inputs

GPUs
tok/s
%
tokens
tasks/h
Result
—
Calculated result
Usable token capacity—
Tasks per second—
Tasks per hour—
Demand coverage—

1. Enter fleet size

Provide the number of GPU devices available to the model-serving workload.

2. Use representative throughput

Enter measured token throughput per GPU for the actual model and serving configuration.

3. Reserve operating headroom

Set the target utilization below 100% to leave room for variation and operational work.

4. Enter task size

Use combined input and output tokens per completed task.

5. Add peak demand

Enter expected tasks per hour to compare capacity with workload.

6. Review capacity

Check both tasks per hour and demand coverage; coverage below 100% indicates a likely queue buildup.

Usable token capacity (tokens/s) = GPUs × Throughput per GPU × Utilization %
Tasks per second = Usable token capacity ÷ Tokens per task
Tasks per hour = Tasks per second × 3,600
Demand coverage (%) = Tasks per hour ÷ Peak demand × 100

The formula assumes the entered throughput already reflects batching, sequence lengths, and serving overhead for a representative workload.

What the result means

The main result is sustainable completed tasks per hour.

Use measured workload data and current provider or hardware benchmarks when available; production results may vary.

Given: 12 GPUs, 1,800 tokens/s per GPU, 70% utilization, 6,100 tokens/task, and 9,000 tasks/hour of peak demand.

Calculation: Usable capacity = 12 × 1,800 × 0.70 = 15,120.0 tokens/s. Throughput = 15,120.0 ÷ 6,100 = 2.479 tasks/s, or 2.479 × 3,600 = 8,923 tasks/hour. Coverage = 8,923 ÷ 9,000 × 100 = 99.1%.

Result: The fleet can sustain about 8,923 tasks/hour, covering 99.1% of the stated peak demand.

Why use target utilization instead of 100%?

Running at full measured capacity leaves little room for bursts, variance, maintenance, or slow requests. A lower target provides operational headroom.

What happens when demand coverage is below 100%?

Requests arrive faster than the estimated completion rate, so the queue is likely to grow until demand falls or capacity increases.

Should retries be included in tasks per hour?

Yes. Add the expected retry rate to workload demand or increase tokens per task to reflect repeated processing.

Can I use CPU or accelerator capacity instead of GPUs?

Yes, provided the throughput value is measured per unit of the hardware you count. Rename the interpretation mentally from GPUs to serving devices.

How is this different from a latency estimate?

Throughput measures aggregate completion capacity, while latency measures how long one task takes. A system can have high throughput and still have unacceptable per-task latency.