toolbox
Home / AI, Data & Technology / Forecast Accuracy Sample Size Estimator

Forecast Accuracy Sample Size Estimator

The Forecast Accuracy Sample Size Estimator calculates the approximate number of forecast-error observations needed to estimate a mean accuracy metric to a chosen margin of error. It is intended for metrics that can be treated as an approximately continuous error measure—such as absolute percentage error, absolute error, or another repeated loss value—when you have an estimate of the metric’s standard deviation.

Enter the expected standard deviation of the forecast-error metric, the desired margin of error around its mean, and a confidence level. The estimator applies the standard normal sample-size formula for a mean and rounds the result up to the next whole observation. The quality of the estimate depends strongly on the supplied standard deviation and on whether the error observations are reasonably independent and representative of the forecasting process you intend to evaluate.

Forecast evaluation precision inputs

Result
—
Required forecast-error observations
Unrounded sample size—
Critical z—
SD / margin ratio—
Approx. total interval width—

1. Choose the accuracy metric
Use a repeated numeric forecast-error metric whose mean you plan to estimate. Keep the same units for the standard deviation and margin.

2. Estimate variability
Enter the expected standard deviation from historical forecast errors, pilot data, or a defensible planning assumption.

3. Set precision
Enter the maximum desired margin of error around the estimated mean accuracy metric.

4. Select confidence
Choose 90%, 95%, or 99%. A higher confidence level increases the required number of observations.

5. Round up operationally
The calculator already rounds the statistical requirement upward. Add more observations if you expect exclusions, missing actuals, dependence, or subgroup analysis.

Required n = (z × σ / E)² Rounded sample size = ceil(n)

z is the normal critical value, σ is the expected standard deviation of the forecast-error metric, and E is the desired margin of error for its mean. σ and E must use the same units. This is the classic large-sample planning formula for estimating a mean with known or planned variability.

What the result means

The result is the approximate number of forecast-error observations needed so the mean metric has the selected normal-approximation margin of error.

Time-series forecast errors are often autocorrelated or seasonal. Effective information can therefore be lower than the raw observation count, so dependence may require a larger evaluation window.

Given: expected forecast-error metric SD = 12 units, desired margin of error = 2 units, 95% confidence.

Calculation: n = (1.96 × 12 / 2)² = 11.76² = 138.30.

Result: Round up to 139 forecast-error observations.

Under the approximation, about 139 comparable observations are needed to estimate the mean error metric within ±2 units at the selected confidence level.

What standard deviation should I enter?

Use the standard deviation of the same forecast-error metric whose mean you want to estimate. Historical out-of-sample errors or a pilot evaluation are preferable to an arbitrary guess.

Can I use this for MAPE?

Yes as an approximation if you treat individual absolute percentage errors as a continuous metric and have a representative standard deviation. MAPE can be unstable when actual values are near zero, so the metric itself may need reconsideration.

Why does halving the margin of error require many more observations?

Sample size varies with the inverse square of the margin. Halving E multiplies the required n by about four, all else equal.

Do overlapping forecast horizons count as independent observations?

Not necessarily. Forecast errors can share information or be serially correlated, which reduces effective sample information relative to a simple independent-observation assumption.

Is this the sample size needed to compare two forecasting models?

Not specifically. This estimator targets precision for one mean error metric. Paired model comparisons usually require a power calculation based on the distribution of paired error differences.