toolbox
Home / AI, Data & Technology / A B Experiment Statistical Power Calculator

A B Experiment Statistical Power Calculator

Estimate the statistical power of a planned two-variant A/B test for a binary conversion metric. Given the control conversion rate, a target relative lift, the sample size per variant, and a two-sided significance level, the calculator approximates the probability that the test will detect the specified difference. Power is most useful before or during experiment planning. It shows whether the available traffic is likely to distinguish the effect you care about from sampling noise, while also making clear that a low-powered design may miss a real improvement even when the experiment is implemented correctly.

Inputs

%
%
users
%
Result
—
approximate statistical power
Target variant rate—
Absolute effect—
Critical z—

1. Enter the control rate
Use the expected baseline conversion proportion.

2. Specify the effect
Enter the relative lift you want the design to be able to detect.

3. Enter sample per variant
Use the planned analyzed sample after expected exclusions, not raw impressions if those are not the unit of analysis.

4. Set significance
Use the same two-sided alpha planned for the final hypothesis test.

5. Interpret power
Compare the estimated power with your experiment-design requirement and reconsider sample size or detectable effect if needed.

Power ≈ Φ((|p2−p1| − z(1−α/2)·SE0) / SE1)

SE0 = √(2p̄(1−p̄)/n) is the null-based standard error for equal groups, and SE1 = √([p1(1−p1)+p2(1−p2)]/n) is the alternative standard error. Φ is the standard normal cumulative distribution.

The approximation is intended for binary outcomes, equal sample sizes, independent observations, and a fixed-horizon two-sided test. Exact or simulation-based power can differ, especially for rare events or small samples.

What the result means

Use the main result together with the supporting statistics and the stated assumptions; it is a planning estimate rather than a guarantee.

Keep units and the unit of analysis consistent. Recalculate when traffic patterns, rates, priors, sample sizes, or design assumptions change.

Given

  • Baseline conversion: 10%
  • Target relative lift: 10%
  • Sample: 15,000 per variant
  • Two-sided significance: 5%

Calculation
p1 = 0.10, p2 = 0.11, and the absolute effect is 0.01. Using the normal approximation gives power close to 80%.

Result
Approximately 80% power

Interpretation
A design near 80% power has about an 80% chance of producing statistical significance if the modeled 10% relative lift is the true effect and assumptions hold.

What does 80% power mean?

Under the modeled effect and assumptions, repeated experiments of this design would achieve statistical significance about 80% of the time. It is not the probability that the current hypothesis is true.

Can power be increased without adding users?

A larger true effect or a higher significance level increases calculated power, but changing alpha alters the false-positive risk. Design changes should be chosen before looking at outcomes.

Why does baseline conversion affect power?

For binary metrics, sampling variance depends on the conversion probability. The same percentage-point effect can have different signal-to-noise ratios at different baselines.

Is post-hoc power useful after I know the observed effect?

Observed-effect post-hoc power often restates information already contained in the test statistic and can be misleading. Confidence intervals and the estimated effect are usually more informative after the experiment.

Does this support unequal traffic splits?

No. The calculator assumes equal sample sizes in control and variant. Unequal allocation requires a formula with separate group sizes.