P-value

p value · significance level · statistical p-value

Definition

A p-value is the probability of obtaining the observed data, or more extreme data, when the null hypothesis is true; it is the basis for deciding whether to reject that hypothesis in an A/B test.

What a p-value is and how to read it

A p-value is the probability of obtaining the observed result, or a more extreme one, assuming the
null hypothesis is true. In an A/B test the null hypothesis is that there is no difference between
variation A and variation B.

Formally: p = 0.03 means that if no difference existed, we would see an effect this large or larger
by chance in 3% of repeated experiments.

What a p-value does not mean:

96% probability that variation B is better — no
The null hypothesis is false with 96% probability — no
The effect is practically meaningful — no, only statistically detectable

Three things that break the interpretation

Peeking — checking interim results with the option to stop. Every additional check is
effectively another test on the same data, which inflates the real error rate. Twenty checks at 5%
risk each puts the chance of a spurious p < 0.05 at roughly 64%.

A small sample — with insufficient data a high p-value does not mean no effect, only not enough
data to detect one. That is a type II error.

Multiple hypotheses — testing 20 metrics at once at p < 0.05 means one of them will look
significant by chance on average. Bonferroni and other multiple-comparison corrections reduce that
risk.

Frequentist versus Bayesian: the real difference

Property	Frequentist (p-value)	Bayesian (probability to be best)
What it measures	Probability of the data under H₀	Probability that B beats A
Interpretation	Technical, frequently misread	Direct
Early stopping	Invalid	Valid
Sample size	Must be fixed in advance	Flexible

Important: a p-value is not a probability of winning. They are different quantities. The
Bayesian approach reports the probability of winning directly, which is both easier to read and
safer when results are being watched during the test.

Frequently asked questions

Does a p-value of 0.04 mean the variation won with 96% probability?

No, and this is the most common misreading. A p-value of 0.04 means that if the null hypothesis holds — no difference — a result this extreme would appear by chance 4% of the time. It is not the probability that the variation is better. That is a different quantity, and it is what the Bayesian approach reports.

Why is the threshold 0.05?

Historically it is a convention introduced by Fisher in the 1920s. It corresponds to a 5% risk of a type I error, a false positive. Medical research often demands 0.01 or 0.001. In product A/B testing 0.05 is the standard, though some teams work at 0.10 for faster decisions with the risk consciously accepted.

How does peeking affect the p-value?

With 20 interim checks, the probability of seeing a p-value below 0.05 at least once is around 64% even when there is no real difference. Each check consumes part of the allowed error budget. Sequential testing or the Bayesian approach are the correct answers when results must be watched early.

How does the Bayesian approach differ?

The Bayesian approach reports the probability that a variation is better than the control instead of a p-value. That quantity is directly interpretable, remains valid at any point in the test and does not require the sample size to be fixed in advance.

Gravity Field

A/B Test Statistics in Gravity Field

Gravity Field defaults to the Bayesian approach — probability to be best instead of a p-value — so results can be read correctly at any point in a test.

✦Bayesian statistics: probability to be best instead of a p-value

✦Built-in protection against peeking through the Bayesian framework

✦A frequentist mode available for teams that prefer p-values

✦Automatic significance detection once the sample is reached

Explore the platform →