Bayesian Statistics

Bayesian Testing · Bayesian A/B Testing · Bayesian Inference

Definition

Bayesian statistics in A/B testing expresses the result as the probability that a variation beats the control, which lets you decide as data accumulates instead of waiting for a fixed p-value threshold.

Bayes theorem and updating beliefs

Bayesian statistics rests on a simple idea: you start with a belief (the prior), you observe data, and
you update that belief in line with the new evidence (the posterior).

Applied to an A/B test:

Prior (before the test): “conversion is usually 2–3%”
Data (from the test):    control CR = 2.1%, variation CR = 2.5%, sample 10K
Posterior:               “variation B beats the control with 91.7% probability”

None of this requires a p-value threshold or a sample size fixed in advance.

The Bayesian approach in A/B testing

In e-commerce A/B testing, the Bayesian model on the beta distribution has become the standard for
product teams. Instead of a binary significant / not significant you get:

Metric	Frequentist	Bayesian
Primary result	Is p-value < 0.05?	Probability to be best = X%
Stopping early	Breaks the statistics	Allowed
Expected loss	Not calculated	Can be calculated
Intuitiveness	Low	High

Expected loss — the expected cost of choosing the wrong variation — is a particularly useful
number: it lets you make a good-enough decision even at 85% confidence, as long as the cost of an
error is small.

The beta distribution for conversion metrics

The beta distribution Beta(α, β) is the natural model for conversion metrics, since CR is a value
between 0 and 1. Its parameters update as data accumulates:

At the start:                     Beta(1, 1) — uniform, “we know nothing”
After 100 conversions and 4,900 non-conversions: Beta(101, 4901) — CR ≈ 2%

Comparing two beta distributions — control vs variation — gives the probability that one value is
greater than the other, and that is exactly the probability to be best.

MAB as Bayesian optimization

Multi-armed bandit works on the same principle: each variation is described by a beta distribution,
and the Thompson sampling algorithm picks the variation for the next user with a probability
proportional to its current posterior. Traffic flows automatically toward the better variations.

Tip: the Bayesian approach does not remove the need to plan sample size. Set a minimum test
duration (two weeks) and a minimum traffic volume per variation — even a high probability after
100 users is not enough data for a production decision.

Frequently asked questions

What is the main difference between Bayesian and frequentist statistics?

Frequentist: if the true difference were zero, we would have observed a result like this in fewer than 5% of cases (p < 0.05). Bayesian: there is a 94.5% probability that variation B is better than the control. The Bayesian result is readable without a statistics background. The frequentist one is stricter about controlling the false-positive rate at a fixed sample size.

Can you look at a Bayesian test before it finishes?

Yes, and that is one of its key practical advantages. The Bayesian approach does not assume a sample size fixed in advance, so interim looks do not break the statistics. Stopping very early on thin data is still a risk — set a minimum sample threshold.

What is a prior and how do you choose one?

The prior is your assumption about the metric before the test starts. An uninformative prior, Beta(1,1), means you know nothing. An informative prior uses historical conversion data. For e-commerce A/B testing, a beta distribution parameterised from historical conversion is standard practice.

At what probability-to-be-best threshold should a test be stopped?

It depends on the cost of being wrong. Industry standards: 95% for changes with moderate risk such as a UI element or a banner; 98–99% for changes with a high cost of error such as a pricing algorithm or a core recommendation algorithm. Always read expected loss alongside it — the expected cost of picking the wrong variation.

Gravity Field