toolbox
Home / AI, Data & Technology / Bayesian Experiment Statistical Power Calculator

Bayesian Experiment Statistical Power Calculator

Estimate the probability that a Bayesian A/B experiment will produce strong evidence that the variant conversion rate exceeds the control rate, under a specified true lift, prior, sample size, and posterior-probability decision threshold. The calculation uses a normal approximation to the two Beta posteriors at the expected data counts. The output is a planning proxy rather than a universal definition of Bayesian power. It is most useful for comparing candidate sample sizes under a fixed decision rule, such as declaring a winner when the posterior probability that the variant is better exceeds 95%.

Inputs

%
%
users
α
β
%
Result
—
approximate probability of meeting decision rule
Posterior P(variant > control)—
Expected variant rate—
Decision threshold—

1. Enter expected control rate
Use the rate you expect for the control population.

2. Specify the expected lift
Enter the relative improvement the variant would have under the planning alternative.

3. Set sample per variant
Use equal analyzed group sizes for this approximation.

4. Define the shared prior
Enter Beta alpha and beta applied independently to both variants before observing data.

5. Choose the decision threshold
Set the posterior probability required to call the variant better, then review the estimated chance of meeting that rule.

Planning probability ≈ Φ((Expected posterior difference − decision z × posterior-difference SE) ÷ sampling SE of observed difference)

At the expected counts, each group has a Beta posterior formed by adding conversions to prior alpha and non-conversions to prior beta. The posterior probability that variant exceeds control is approximated from the difference of posterior means and variances. The outer planning probability approximates how often sampling noise would still clear the chosen posterior-probability threshold.

This is an analytical approximation to a decision-specific Bayesian operating characteristic, not an exact simulation. Exact planning should simulate binomial data and posterior decisions, especially with strong priors, small samples, rare outcomes, asymmetric losses, or stopping rules.

What the result means

Use the main result together with the supporting statistics and the stated assumptions; it is a planning estimate rather than a guarantee.

Keep units and the unit of analysis consistent. Recalculate when traffic patterns, rates, priors, sample sizes, or design assumptions change.

Given

  • Expected control: 10%
  • Expected lift: 10% relative, so variant: 11%
  • Sample: 15,000 per variant
  • Prior: Beta(1,1)
  • Decision rule: posterior probability variant > control ≥ 95%

Calculation
At the expected counts, the Beta posteriors center near 10% and 11%. Their difference is compared with the posterior uncertainty implied by the 95% decision threshold, then sampling variation is used to approximate how often that threshold will be exceeded.

Result
A decision probability estimate near the planned operating characteristic

Interpretation
Use the result to compare sample-size scenarios under the same prior and decision threshold; simulation is preferred for final design validation.

Is Bayesian statistical power the same as frequentist power?

Not necessarily. Bayesian designs usually define an operating characteristic for a specific posterior decision rule. This calculator reports an approximation for one such rule.

What does the posterior probability threshold control?

It sets how much posterior evidence is required before declaring the variant better. Raising the threshold makes the rule more conservative and generally reduces the chance of declaring a winner at a fixed sample size.

Why do both groups use the same prior?

The calculator assumes a common Beta prior shape for simplicity and symmetry. If you have justified group-specific priors, simulation with separate priors is more appropriate.

Can I use optional stopping with this result?

The displayed approximation assumes a fixed sample size. Bayesian monitoring can support different stopping rules, but their operating characteristics should be evaluated under the actual sequential design.

When should I simulate instead of using this calculator?

Use simulation when priors are strong, rates are rare, sample sizes are small, the decision rule includes practical-effect thresholds or losses, or the experiment can stop early.