Statistical Power

Power · Test Power · 1 minus beta

Definition

Statistical power (1 − beta) is the probability that a test detects a real effect when one exists; the industry standard is no lower than 80%.

What power is

Power (1 − beta) is the probability that a test correctly detects a real effect. Its complement,
beta, is the probability of a type II error: missing a genuine improvement.

Reality:              B is better         B is not better
Test says "better":   Correct (power)     Type I error (alpha)
Test says "no":       Type II error (beta) Correct

The industry standard is power at 80% or above (beta at 20% or below) with alpha at 0.05. Out of 100
tests where B really is better, 80 are correctly identified and 20 are missed.

What drives power

Sample size. The main lever. More users in the test means less random noise and a better chance
of seeing a real effect.

Effect size (MDE). A large effect (+20% CR) is easier to detect at the same sample than a small
one (+3%). That is why a realistic estimate of the minimum meaningful effect belongs in the plan.

The baseline metric. At a 5% conversion rate the variance is lower than at 0.5%, so a smaller
sample achieves the same power.

Calculating the sample you need

For a standard test at alpha 0.05 and 80% power the formula needs:

the baseline conversion rate
the MDE, the minimum detectable effect, say 10%

Approximate relationships:

Baseline CR	MDE	Traffic per variation
2%	15%	~6,500
2%	10%	~14,700
1%	15%	~13,000
1%	10%	~29,400

Important: these numbers are per variation. With two variations the total is double.

Power and Bayesian statistics

Power in the classical sense is a frequentist instrument. Bayesian statistics do not require power to
be fixed in advance: evidence accumulates in favour of each hypothesis instead. Probability to be
best rises as data arrives, with no peeking risk and no formula-driven sample plan.

For teams with limited traffic, the Bayesian route allows a grounded decision earlier — when the
data is still short of 80% frequentist power but already sufficient for a confident probabilistic
read.

Frequently asked questions

What does 80% power mean?

If a real effect exists — variation B genuinely is better than A — a test with 80% power will detect it in 80% of cases. In the other 20% the test misses the improvement and keeps the control. That is a type II error, a false negative.

How is power related to sample size?

Directly. A larger sample means higher power at the same MDE. Raising power from 80% to 90% requires roughly 35% more sample. That is precisely why a sample size calculator treats power as a first-class parameter.

What is the difference between power and statistical significance?

Significance (the p-value threshold) governs the probability of declaring a winner that does not exist — a type I error. Power governs the probability of finding a winner that does exist — protection against a type II error. A good test controls both: usually alpha at 0.05 and power at 80% or above.

Why are e-commerce tests so often underpowered?

Limited traffic. At a 1% conversion rate and 5,000 visitors a day, a 10% MDE needs 21 days or more at 80% power — an unrealistic wait for a team that wants answers sooner. The consequence is a stream of tests at 40–60% power that regularly fail to see real effects.

Gravity Field

Test Power in Gravity Field

Gravity Field takes a Bayesian approach to A/B testing, where power is assessed through a continuously updated probability to be best rather than a fixed horizon and a preset power threshold.

✦Bayesian statistics — probability to be best instead of a classical p-value

✦Autopilot (MAB) — traffic shifts toward the better variation as data arrives

✦A sample size calculator that accounts for MDE and target power

✦Real-time monitoring of accumulated data without the peeking problem

Explore the platform →