What a control group is
Every A/B test has at least two segments: the control group (group A) sees the current version
of the page or algorithm, the treatment group (group B) sees the changed one. The difference in
metrics between them is the measurable effect of the change.
Without a control group, any observed lift could equally be explained by external factors: the start
of a season, a new ad campaign, or plain random fluctuation in traffic.
What a valid control group requires
- Simultaneous traffic accumulation with the treatment group — sequential periods are not comparable
- The same sources — control and treatment must draw from the same channels
- Immutability — the control variant must not change during the test
- Sticky assignment — one visitor always lands in the same group on every visit
Important: if the same person sees different variants across sessions, the data is contaminated
and the conclusions are unreliable.
Sizing the control group
A 50/50 split is a sensible default, but not the only one.
| Scenario | Recommended split |
|---|---|
| Classic A/B test | 50/50 |
| A/B/C test (three variants) | 34/33/33 |
| Expensive change with regression risk | 80/20 (80% control) |
| Bandit autopilot | Dynamic, control at 10% or more |
| Long-running holdout | 90/10 (10% in holdout) |
Control group versus holdout
A control group is the baseline inside one specific test. A holdout group extends the idea: a
segment deliberately excluded from all personalization for a long period, three to six months. The
holdout measures the cumulative effect of every change rather than the effect of one experiment.
Common mistakes
- Changing the control during the test — updating the underlying page template mid-test
invalidates the result - Unbalanced traffic — if advertising drives traffic only to the treatment page, the group
allocation is wrong - A control group that is too small — under aggressive allocation to the apparent winner, the
control may not gather enough traffic for reliable conclusions