Test Duration

The two factors behind duration

Duration is set independently by two parameters, and the real duration is the larger of the two.

1. The statistical factor — how many users the calculation requires, which depends on the MDE,
the significance level and the power.

2. The temporal factor — at least two full weeks regardless of traffic. That covers at least two
complete business cycles (weekdays plus weekends, twice).

Traffic: 20,000 unique visitors per week
Baseline CR: 2.5%
MDE: 10% relative → 2.75%
Required sample: ~65,000 per variation

Computed duration: 65,000 / 10,000 = 6.5 weeks
Minimum duration: 2 weeks
Result: 6.5 weeks, round to 7

Why whole weeks matter

Shopper behaviour differs substantially by day of week. Weekends often convert higher in some
categories and lower in others. A test started on a Wednesday and stopped the following Monday
contains two weekends in one group and one in the other — a structural bias in the sample.

Rule: start a test early in the week (Monday or Tuesday) and schedule the stop at the same time
N whole weeks later.

Duration and the bandit

A multi-armed bandit has no fixed duration in the classical sense — traffic shifts dynamically
toward the better variation. Even so, a minimum period of 7–14 days before aggressive reallocation
is advisable, to avoid the novelty effect — the short-lived metric lift any new variant enjoys.

Duration at low traffic

Below 3,000–5,000 unique visitors a week, classical A/B testing loses its meaning — the test would
take months. In that situation, ship the change on expert judgement and follow it with cohort
analysis.