Statistical Significance

Significance Level · Statistical Confidence · Significance

Definition

Statistical significance is the degree of confidence that the difference between A/B test variations is real rather than the product of random fluctuation; the usual requirement is p < 0.05.

What statistical significance is

Statistical significance answers one question: could the difference observed between variations have
come up by chance? If the probability of chance is low, the result counts as significant.

Mathematically this is expressed through the p-value. p < 0.05 means that, assuming there is no real
effect, the probability of getting a divergence this large or larger is under 5%.

Variation A: CR = 2.10% (n = 12,400)
Variation B: CR = 2.47% (n = 12,500)
Difference:  +0.37 pp, lift +17.6%
p-value = 0.031  →  significant at α = 0.05

Important: significance confirms that the difference is reliable, not that it matters. A
difference can be statistically significant and commercially insignificant at the same time.

Two approaches: frequentist and Bayesian

Parameter	Frequentist	Bayesian
Primary metric	p-value	Probability to be best
Threshold	p < 0.05 (α = 0.05)	≥ 95% probability
Stopping early	Not allowed	Allowed
Interpretation	Technical	Intuitive

For e-commerce the Bayesian approach is preferable: it lets you stop a test early and correctly when
there is a clear winner, and it does not require the marketing team to understand what a p-value is.

Significance level and statistical power

Two key parameters are set when an experiment is planned:

Significance level (α) — the acceptable probability of a false positive, a type I error. The
standard is α = 0.05.
Statistical power (1 − β) — the probability of detecting a real effect when one exists. The
standard is 80%.

Both are fixed before the test starts and they directly determine the sample size required. Moving
the thresholds after the fact to push a result through is data manipulation.

Significance and MDE: how they connect

Statistical significance is inseparable from the minimum detectable effect (MDE). If the expected CR
lift is 5%, you need one number of participants. If it is 2%, you need several times more. On thin
traffic a test will finish with a non-significant result even when the real effect exists — that is a
type II error.

Common mistakes

Stopping on the first good-looking number (peeking). Across 20 interim checks, the odds of
seeing p < 0.05 by chance exceed 60%.
Recomputing the threshold after the fact. A result that is almost significant is a
non-significant result.
Confusing significance with effect size. A CR lift of +0.1% can be significant and still not
worth shipping.
Ignoring SRM. If the actual split between groups differs from the configured one, sample ratio
mismatch invalidates the significance calculation entirely.

Frequently asked questions

What does p-value < 0.05 mean?

The p-value is the probability of getting the observed difference in metrics, or a more extreme one, if there is no real effect (the null hypothesis holds). p < 0.05 means a difference like this comes up by chance in fewer than 5 cases out of 100. It does not guarantee the effect is real, but it is the standard threshold for making decisions.

Significance is reached — can the test be stopped?

Stopping a test the first time p drops below 0.05 is the classic peeking mistake. With repeated interim checks the odds of a false positive climb sharply. Wait for the calculated sample size, or use sequential testing methods designed for continuous monitoring.

How is statistical significance different from practical significance?

Statistical significance speaks to the reliability of a difference, not its scale. A test on 500K people can show a significant CR lift of 0.02% — a real effect, and a commercially meaningless one. Practical significance is set by the minimum detectable effect (MDE) defined before the test.

Is the Bayesian approach the same thing as significance?

No. The Bayesian approach replaces the p-value with probability to be best. The threshold is usually 95–99% and the reading is more intuitive: there is a 97% probability that variation B is better than A. It also allows a test to be stopped early and correctly once the threshold is met.

Gravity Field