Feature Store Statistical Power Calculator

1. Enter the baseline rate
Use the current feature-store pass, freshness, retrieval, or other binary success rate.

2. Define the minimum lift
Enter the smallest absolute percentage-point increase that would matter operationally.

3. Set samples per group
Use the planned number of independent observations in both the baseline and comparison groups.

4. Choose significance
Select the Type I error threshold used by your two-sided test.

5. Review power
Higher power means the planned test is more likely to detect the stated lift if that lift is real.

p₂ = p₁ + liftSE₀ = sqrt(2 × p̄ × (1 − p̄) / n)SE₁ = sqrt((p₁(1 − p₁) + p₂(1 − p₂)) / n)Power ≈ Φ((|p₂ − p₁| − zα × SE₀) / SE₁)

Where:

• p₁ = baseline proportion
• p₂ = alternative proportion after the absolute lift
• p̄ = average of p₁ and p₂
• n = sample size per group
• zα = two-sided normal critical value
• Φ = standard normal cumulative distribution

Assumptions: The two groups are treated as independent, equally sized, and adequately represented by a normal approximation. This is a planning estimate rather than a substitute for the exact test procedure used in analysis.

What the result means

Power is the approximate chance of rejecting the no-difference hypothesis when the specified alternative rate is true.

Very small samples, rare outcomes, unequal group sizes, repeated observations, or multiple-testing procedures can require a different power calculation.

Given: 92% baseline success, a +2 percentage-point detectable lift, 2,200 observations per group, and a 5% significance level.

Calculation: p₁ = 0.92 and p₂ = 0.94. The calculator forms the pooled and alternative standard errors, then compares the 0.02 difference with the two-sided 1.959964 critical value.

Result: Approximate statistical power is 75.2% with 4,400 total observations.

Interpretation: If power is below your study requirement, increase the sample size or reconsider the minimum effect you need to detect.