How an A/B test works
A classic test has two versions: the control (A), which is the current experience, and the
variant (B), which carries the change. Visitors are assigned at random through a cookie or a
server-side binding — once a user lands in a group, they always see the same version.
Once the sample is large enough, the results are compared on a key metric. If the difference is
statistically significant (typically p < 0.05, or a probability to be best of 95% or higher), the
winner ships as the new default.
CR(A) = 2.1% → CR(B) = 2.5% Lift = +19%
RPV(A) = $6.40 → RPV(B) = $7.50 Lift = +17%
Tip: use RPV (revenue per visitor) as your primary metric — it captures conversion rate and
order value at the same time.
Frequentist vs Bayesian: two approaches
| Parameter | Frequentist | Bayesian |
|---|---|---|
| Primary metric | p-value | Probability to be best |
| Stopping early | Not allowed | Allowed |
| Works with MAB | ✗ | ✓ |
| Speed of decision | Slower | Faster when the gap is large |
For e-commerce the Bayesian approach is the better fit: it lets you react sooner and it is natively
compatible with automatic traffic allocation (MAB). Gravity Field uses Bayesian statistics by default.
Autopilot: Multi-Armed Bandit (MAB)
Multi-armed bandit is an extension of A/B testing. Instead of a fixed 50/50 split, the algorithm
reallocates traffic dynamically toward the winning version while the test is still running. That
cuts the conversions lost to the losing variant by 30–50% compared with a classic split.
A/B tests for recommendations and personalization
In e-commerce, A/B tests are most often run not on UI elements but on recommendation strategies:
comparing algorithms, widget configurations and the placement of recommendation blocks. The key
metric is attributed revenue, measured against an attribution window of 7–14 days.
Common mistakes
- Peek-and-stop: calling the test the moment the numbers look good is the fastest route to a
false positive. Across 20 interim checks, the odds of seeing p < 0.05 by chance exceed 60%. - Several variables at once: change the headline, the button colour and the photo together and
you can no longer attribute the result to any of them. - Too small a sample: at a 1% CR and 10K visitors a day, detecting a 15% MDE takes roughly 17–21 days.
- Ignoring seasonality: a test run in the week before a major sale draws an unrepresentative audience.