toolbox
Home / AI, Data & Technology / A B Experiment Sample Size Estimator

A B Experiment Sample Size Estimator

Estimate the sample size needed per variant for a two-group A/B test with a binary conversion outcome. The calculator uses a normal-approximation formula based on the baseline conversion rate, the minimum detectable relative lift, the significance level, and the desired statistical power. This is a planning estimate for independent groups of similar size. It helps experiment owners translate an effect they care about into a traffic requirement before launch, making it easier to judge whether a test is feasible within the available audience and time window.

Inputs

%
%
%
%
Result
—
required sample per variant
Total sample—
Target variant rate—
Absolute effect—

1. Enter the baseline rate
Use the best estimate of the control conversion rate for the same metric and population.

2. Define the minimum lift
Enter the smallest relative improvement that would be worth detecting.

3. Set significance
Choose the two-sided Type I error rate used for the hypothesis test.

4. Choose power
Set the probability of detecting the specified effect when it is truly present.

5. Review traffic need
Use the per-variant and total sample estimates to assess test duration and feasibility.

n ≈ [(z(1−α/2)√(2p̄(1−p̄)) + z(power)√(p1(1−p1)+p2(1−p2)))²] ÷ (p2−p1)²

p1 is the baseline conversion proportion, p2 = p1 × (1 + relative lift), and p̄ = (p1 + p2) / 2. z() is the standard normal quantile. The displayed n is rounded up for each variant.

This approximation assumes independent observations, a binary outcome, equal allocation, and sufficiently large samples for the normal approximation. Sequential testing, clustering, multiple comparisons, or unequal allocation require different planning methods.

What the result means

Use the main result together with the supporting statistics and the stated assumptions; it is a planning estimate rather than a guarantee.

Keep units and the unit of analysis consistent. Recalculate when traffic patterns, rates, priors, sample sizes, or design assumptions change.

Given

  • Baseline conversion: 10%
  • Minimum detectable relative lift: 10%
  • Two-sided significance level: 5%
  • Power: 80%

Calculation
p1 = 0.10 and p2 = 0.11. Using z0.975 ≈ 1.96 and z0.80 ≈ 0.842 gives approximately 14,744 observations per variant.

Result
About 14,744 users per variant, or 29,488 total

Interpretation
The experiment needs roughly this many independent observations to have 80% power to detect a 10% relative lift from a 10% baseline under the stated assumptions.

Why is the required sample large for small lifts?

Small differences are harder to distinguish from random variation, so the required sample grows quickly as the minimum detectable effect shrinks.

Is the lift relative or absolute?

The lift input is relative. A 10% lift on a 10% baseline targets 11%, which is a 1 percentage-point absolute increase.

Can I use this for revenue per user?

Not with this binary-proportion formula. Continuous metrics need variance information and a sample-size method designed for means or other outcome distributions.

Does the estimate account for peeking or sequential stopping?

No. It assumes a fixed-horizon design. Sequential methods need boundaries or alpha-spending rules that change the sample planning.

Should I round the sample size?

Yes. The calculator rounds up per variant because partial observations are not possible, and practical tests often add extra traffic for exclusions or data loss.