What a holdout is for
A standard A/B test measures the effect of one specific change over a short period. A real
personalization programme is dozens of consecutive experiments over a year. Summing their results
naively does not work: changes interact, some effects decay, the novelty of the first weeks fades.
A holdout group solves that. It is a slice of the audience frozen in the original state for the
whole period. Comparing its metrics against everyone else six months later gives an honest answer
to the question of what the personalization programme actually delivered.
How a holdout works in practice
Traffic
├── 90% — regular users
│ receive every personalization change
│ participate in A/B tests
└── 10% — holdout group
receive no personalization
participate in no tests
see the baseline site for the whole period
After six months:
Holdout RPV: 4.20
Main base RPV: 4.87
Aggregate effect: +16%
Technical requirements
Sticky assignment: a user who lands in the holdout has to stay there the whole time. Switching
between groups destroys the integrity of the measurement.
Isolation from every experiment: if the holdout is accidentally swept into a few A/B tests, the
data is spoiled. The system has to check holdout membership before enrolling anyone in any test.
Sufficient size: with a 5% holdout and a 2% baseline conversion rate, detecting a 5% cumulative
effect can take six to twelve months depending on traffic.
Holdout versus a continuous bandit
A multi-armed bandit continuously reallocates traffic toward the better variation, which is
incompatible with a holdout in the classic sense. That is a genuine trade-off: the bandit monetises
the winner faster, and gives up the ability to measure the long-run accumulated effect honestly.
Tip: if an organisation is running a large personalization programme, a holdout group is worth
the potential revenue cost on 5–10% of traffic. It is what lets you prove the real return of the
whole programme to a CFO a year later, rather than presenting a pile of disconnected tests.