The AI Agent Latency Capacity Estimator models how long an agent task takes when it combines model calls, tool calls, and repeated attempts. It also translates concurrency into an estimated maximum task rate, making the tradeoff between response time and capacity visible.
Use it for early architecture planning, service-level objective checks, or load-test targets. The estimator reports average end-to-end latency, estimated tasks per minute, model-time share, and the concurrency needed to reach a chosen demand level.
1. Describe the agent loop
Enter average model and tool calls made by one completed task.
2. Add component latency
Use measured or expected average seconds for each model and tool call.
3. Include repeated work
Enter the share of tasks that require another equivalent attempt.
4. Set concurrency and demand
Provide available task slots and an optional target rate.
5. Check latency and capacity
Compare predicted tasks per minute with the target and review required concurrency.
Capacity is the theoretical average task rate if concurrent slots stay busy and the entered latency components are representative.
Queueing, rate limits, network variance, sequential dependencies, and tail latency can reduce real production capacity.
Given: 4 model calls at 1.8 seconds, 2 tool calls at 0.7 seconds, 5% retries, and 30 concurrent slots.
Calculation: Base latency = (4 × 1.8) + (2 × 0.7) = 8.6 seconds. Adjusted latency = 8.6 × 1.05 = 9.03 seconds. Capacity = 30 × 60 ÷ 9.03 = 199.3 tasks per minute.
Result: The configuration supports about 199 tasks per minute under steady average conditions.
The formula treats calls as sequential. For parallel branches, enter the critical-path time rather than summing every branch.
Use average values for throughput planning and p95 values when evaluating a p95 service target.
It means one task can be in progress independently of other tasks.
Provider throttling, queueing, uneven task duration, and idle gaps reduce utilization.
Common levers include fewer model turns, smaller context, faster models, parallel tool execution, and cached results.