What MDE is and how it connects to the other parameters

MDE is not an arbitrary threshold. It is mathematically tied to three other test parameters:

Test parameters:
alpha (significance level) — usually 0.05
beta (type II error rate)  — usually 0.20
1 - beta (power)           — usually 0.80
Baseline CR                — the current conversion rate

From these follows:
n (sample size) ↔ MDE

If alpha, beta and traffic are fixed, the MDE follows. If alpha, beta and the desired MDE are fixed,
the required sample size follows.

How MDE drives test duration

Baseline CR = 2.0%, alpha = 0.05, beta = 0.20

MDE = 20% (CR from 2.0% to 2.4%):  n ≈ 5,000 per group
MDE = 10% (CR from 2.0% to 2.2%):  n ≈ 20,000 per group
MDE = 5%  (CR from 2.0% to 2.1%):  n ≈ 80,000 per group

At 5,000 unique visitors a day and a 50/50 split:

  • MDE 20% → the sample is there in about two days (with a 14-day minimum for a full cycle)
  • MDE 5% → the sample alone takes 32 days or more

Tip: do not set the MDE as low as possible — that is the trap. Ask instead what minimum
conversion lift would justify shipping this change. That number is your MDE.

Practical guidance

Situation Recommended MDE
Homepage, high traffic 5–10%
Product page, medium traffic 10–15%
Category page, low traffic 15–25%
Recommendation widget (on RPV) 5–10%

Common mistakes

  • Lowering the MDE without checking traffic — the test stretches over months and the data ages
  • An MDE below the economically meaningful threshold — even a significant result does not pay back
  • Ignoring MDE while peeking — stopping early on an apparent effect below the MDE produces a
    false positive