Type I Error (False Positive)

type i error · type 1 error · false positive · alpha error · false positive rate

Definition

A Type I error is a false-positive conclusion: a variation is declared better than the control when no real difference exists; the probability of that error is controlled by the threshold α.

What a Type I error is

In A/B testing the null hypothesis (H₀) states that variations A and B are identical. A Type I error
is rejecting H₀ when it is in fact true. Put simply: the test “found a winner” that does not exist.

The parameter α controls the acceptable probability of that error. At α = 0.05, in a test with no
real effect, one experiment in twenty will return a false positive purely by chance.

Practical consequences

For a team running 40 tests a year at α = 0.05:

Expected number of false "wins" = 40 × 0.05 = 2 per year

That means two rollouts of changes that do not actually improve the product. With a small MDE
(minimum detectable effect) and low traffic, the real share of false positives can be higher still.

The main amplifiers of Type I error

Peeking. Check the results daily and stop at the first p < 0.05, and the real probability of a
Type I error across 20 interim checks exceeds 30% — against a stated α of 0.05. This is the single
most common mistake in A/B testing.

Multiple metrics without a correction. Evaluate a test on 10 metrics and the probability of a
chance significant result on at least one is already around 40%. The fix is to pick one primary
metric in advance and apply a Bonferroni correction or FDR control to the rest.

Sample Ratio Mismatch (SRM). When the actual group ratio differs from the planned one, the
results are distorted and the significance can be spurious.

Tip: the Bayesian approach does not use a p-value or an α threshold directly — it reports the
probability to be best instead. That does not remove Type I error entirely, but it makes it far
less dependent on how many times you looked at the data.

Type I versus Type II error

	Type I error (α)	Type II error (β)
What happens	We roll out a bad change	We reject a good change
Controlled by	The α level	Test power (1 − β)
Reduced by	A smaller α, sequential testing	More traffic, a smaller MDE
Trade-off	A stricter α raises β	A stricter β raises α

Frequently asked questions

What is the α level and how do you choose it?

α is the acceptable probability of a Type I error. At α = 0.05 we accept that 1 test in 20 with no real effect can return a significant result by chance. For e-commerce the standard is α = 0.05. When the cost of a mistake is high — sweeping algorithm changes, for instance — teams use α = 0.01.

Why does peeking increase the Type I error?

Every interim check carries its own chance of seeing p < 0.05 by accident. Check a test 20 times and the cumulative probability of a false positive is far above the stated α = 0.05. The fixes are sequential testing, a Bonferroni correction, or a Bayesian approach.

How does the Type I error relate to A/A testing?

An A/A test checks whether the platform generates false positives systematically. If an A/A test returns a significant result, there is a structural problem in the system — SRM or faulty randomization — and the real Type I error rate is higher than the stated α.