toolbox
Home / AI, Data & Technology / LLM Batch Processing Throughput Estimator

LLM Batch Processing Throughput Estimator

The LLM Batch Processing Throughput Estimator turns an observed batch run into successful task throughput, token throughput, and a projection for a larger target workload. It uses elapsed wall-clock time, average tokens per task, completion success rate, and worker count to normalize performance.

This page is useful after a pilot run because it starts from measured output rather than theoretical device capacity. Use the same definition of a completed task across runs, and include the full elapsed interval that matters operationally. The projection assumes the observed rate remains stable at the new scale; storage bandwidth, provider limits, queue contention, or a different task-length distribution can break that assumption.

Observed batch performance

tasks
min
tokens
%
workers
tasks
Result
—
successful tasks per second
Successful token throughput—
Successful tasks per worker-hour—
Projected target completion time—
Unsuccessful tasks in observed run—

1. Enter observed completions
Use the total tasks attempted or completed in the measured run, consistently with the success-rate definition.

2. Enter elapsed time
Include the wall-clock interval used to produce the observed task count.

3. Set average token size
Provide average input plus output tokens for successful tasks.

4. Apply success rate
Exclude tasks that did not produce usable output from successful throughput.

5. Enter worker count
Normalize the run into successful tasks per worker-hour.

6. Project a target
Enter a future task count to estimate completion time at the observed rate.

Successful tasks = Observed tasks × Success rate Successful tasks/s = Successful tasks ÷ Elapsed seconds Token throughput = Successful tasks × Tokens per task ÷ Elapsed seconds Projected hours = Target tasks ÷ Successful tasks/s ÷ 3,600

What the result means

The main result is the observed rate of usable completed tasks, not merely attempted tasks.

Estimates depend on the model, serving stack, hardware, quantization, batching strategy, and workload mix. Validate the result with measurements from your own environment before committing capacity or budget.

Given

  • 300,000 observed tasks
  • 95 minutes elapsed
  • 1,250 tokens per task
  • 98.5% success
  • 96 workers

Calculation

Successful tasks = 300,000 × 0.985 = 295,500. Throughput = 295,500 ÷ 5,700 = 51.842 tasks/s. Token throughput = 51.842 × 1,250 = 64,802.6 tok/s.

Result

51.842 successful tasks/s.

At the same rate, one million target tasks would take about 5.36 hours.

Should failed tasks be counted in elapsed time?

Yes. They consumed part of the observed run, so excluding their time would overstate successful throughput.

Can I compare runs with different worker counts?

Use tasks per worker-hour for a rough normalized comparison, but also inspect scale efficiency because worker performance may change as concurrency increases.

Why project from successful throughput?

Many downstream workflows care about usable outputs. Attempt throughput can look healthy even when validation or completion failures create rework.

How should very long tasks be handled?

Segment them from the main workload or use a weighted task mix. A changing tail of long records can make a simple linear projection unreliable.

Is token throughput or task throughput more important?

Token throughput is better for hardware and provider capacity, while task throughput is better for operational deadlines. Review both when task sizes vary.