toolbox
Home / AI, Data & Technology / A B Experiment Error Rate Estimator

A B Experiment Error Rate Estimator

Estimate false-positive and false-negative error rates for a planned A/B decision from the chosen significance level and statistical power. It also converts those rates into expected counts across a batch of experiments when you provide the number of tests and an assumed share of truly null experiments. The calculator is a planning aid for understanding repeated-testing behavior. It separates Type I error (rejecting a true null) from Type II error (missing a real effect), then shows how both may contribute to incorrect decisions across an experiment portfolio under an explicit base-rate assumption.

Inputs

%
%
tests
%
Result
—
expected incorrect decisions
Expected false positives—
Expected false negatives—
Type II error rate—

1. Enter alpha
Use the significance threshold associated with a false-positive decision in each planned test.

2. Enter power
Use the design power for the effect size that matters to the experiment.

3. Set experiment count
Enter the number of comparable decisions you want to summarize.

4. Estimate the null share
Provide an explicit assumption for how many experiments truly have no relevant effect.

5. Review error composition
Compare expected false positives and false negatives rather than relying only on the combined total.

Expected false positives = Tests × null share × α; Expected false negatives = Tests × (1 − null share) × (1 − power)

Alpha is the Type I error probability conditional on the null being true. Beta = 1 − power is the Type II error probability conditional on the modeled alternative being true.

Expected counts are portfolio averages, not guaranteed outcomes for a specific set of experiments. The calculation also assumes the entered alpha and power apply similarly across the tests and does not adjust for multiplicity across a shared decision family.

What the result means

Use the main result together with the supporting statistics and the stated assumptions; it is a planning estimate rather than a guarantee.

Keep units and the unit of analysis consistent. Recalculate when traffic patterns, rates, priors, sample sizes, or design assumptions change.

Given

  • Alpha: 5%
  • Power: 80%
  • Experiments: 100
  • Assumed null share: 70%

Calculation
Expected false positives = 100 × 0.70 × 0.05 = 3.5. Beta = 20%. Expected false negatives = 100 × 0.30 × 0.20 = 6.

Result
About 9.5 incorrect decisions in expectation

Interpretation
Across many similar portfolios, the assumptions imply about 3.5 false positives and 6 missed real effects per 100 experiments on average.

Is alpha the probability that a significant result is false?

No. Alpha is the false-positive rate conditional on the null hypothesis being true. The probability a significant result is false also depends on how often null hypotheses are actually true and on power.

Why does the calculator ask for a null share?

Expected counts across many experiments depend on the underlying mix of true nulls and real effects. Without a base-rate assumption, alpha and beta alone cannot determine portfolio-level error counts.

Does multiple testing change these numbers?

It can. If many related hypotheses form one decision family, family-wise or false-discovery adjustments may change the effective thresholds and error behavior.

Can the expected incorrect-decision total be fractional?

Yes. It is an expectation over repeated portfolios, so values such as 3.5 false positives are meaningful averages even though an actual portfolio has whole-number outcomes.

Is a false negative always less costly than a false positive?

No. The practical cost depends on the decision context. Experiment design may need different alpha, power, or loss-based criteria when the two error types have unequal consequences.