Frequentist Statistics

frequentist statistics · frequentist approach · classical statistics · frequentist ab testing · frequentist inference

Definition

Frequentist statistics is an approach to testing built on the p-value and a sample size fixed in advance, in which probability is read as the long-run frequency of an event across repeated trials.

What frequentist statistics is

The frequentist approach is the classical reading of probability: the probability of an event is the
share of times it occurs when the experiment is repeated many times over. It is an objectivist
statistics — it carries no prior beliefs and rests only on the observed data.

In A/B testing, the frequentist approach runs as a three-step procedure:

Before the test: set α (the significance level, usually 0.05) and β (the acceptable Type II
error, usually 0.2, i.e. 80% power), then compute the required sample size from the formula
During the test: wait, without intervening, until the full sample is collected
After the test: read the p-value and make the call

Key metrics

H₀ (null hypothesis):        CR(A) = CR(B), no difference
H₁ (alternative hypothesis): CR(B) ≠ CR(A)

α = 0.05 (Type I error — a false positive)
β = 0.20 (Type II error — a false negative)
Power = 1 − β = 0.80

p-value < α → reject H₀ → the result is "statistically significant"

Metric	What it means
p-value	The probability of this data given that there is no effect
α (alpha)	The false-positive threshold (standard: 0.05)
Power	The chance of detecting a real effect when one exists
Confidence interval	The range containing the true difference with probability (1 − α)

The core discipline: no peeking

The first rule of frequentist testing is that there are no interim decisions. Look at the results
every day and stop the test the moment the gap looks attractive, and you inflate the Type I error:

Interim looks:          1     5     10     20
Real Type I error:    ~5%   ~14%   ~19%   ~25–30%
(at a nominal α = 0.05)

The fix is either strict discipline with no interim looks at all, or a move to sequential testing
with corrected boundaries.

Frequentist vs Bayesian: choosing between them

Both paradigms answer the same question, by different means:

Criterion	Frequentist	Bayesian
Interpretability for the business	Harder (the p-value is unintuitive)	Easier (“the probability that B is better”)
Early stopping	Requires sequential testing	Native
MAB / autopilot	Not directly compatible	Natively supported
Regulatory requirements	The standard in pharma and finance	Less widely accepted
Reproducibility	High	Depends on the prior

Tip: for most e-commerce experiments the Bayesian approach is the more practical one — it lets
you react sooner, it does not demand rigid peeking discipline, and it supports automatic traffic
allocation natively. Frequentist is preferable when you need formal reproducibility or have to
hand the result to external auditors.

Frequently asked questions

What is a p-value and why p < 0.05?

The p-value is the probability of observing the result you got, or a more extreme one, assuming the null hypothesis is true — that is, assuming there is no difference. P < 0.05 means the chance of seeing a difference this large purely at random is under 5%. The 5% threshold is a historical convention, not a law of nature. For critical decisions teams use α = 0.01.

Why can you not look at the results before the test ends?

That is peeking. Every interim look is effectively an extra test. Checking daily on a 20-day test pushes the real false-positive rate from 5% up to roughly 30–65%. This is known as the multiple testing problem.

When is the frequentist approach preferable to the Bayesian one?

When reproducibility and transparency for stakeholders matter: a p-value and a confidence interval are easier to explain to a board than a posterior probability. When regulatory compatibility matters: in pharma and finance the frequentist framework is the standard. And when the test is run strictly once, with no interim looks.

What is sequential testing in a frequentist context?

Sequential testing is an extension of the frequentist approach that allows a test to be stopped early without breaking the maths. It uses corrected boundaries (Wald's SPRT, mSPRT) that account for the data accumulated so far. It solves the peeking problem without switching to a Bayesian paradigm.