Sample Size

Sample Size · N-Size · Required Sample

Definition

Sample size is the minimum number of observations per A/B test variation that makes it possible to detect the target effect at the chosen statistical power.

Why the sample is calculated in advance

Sample size is not a technical detail, it is the foundation of a correct test. Launch a test and stop
it whenever the result looks appealing, and the probability of a false-positive conclusion rises
sharply. Across 20 interim checks it exceeds 60%, even when there is no real effect at all.

Calculating the sample before launch sets an objective horizon: the test runs exactly as long as it
needs to — no shorter, no longer.

The formula and its key parameters

For a two-proportion test, the sample size per variation is:

n = 2 × (z_α/2 + z_β)² × p̄(1 − p̄) / δ²

z_α/2 = 1.96  (α = 0.05, two-sided test)
z_β   = 0.84  (80% power)
p̄    = (p_A + p_B) / 2
δ     = |p_B − p_A|  (MDE)

Baseline CR	MDE (relative)	n per variation	Days at 5K/day
2%	10%	~37,600	~15
2%	5%	~150,000	~60
5%	10%	~14,200	~6
5%	5%	~56,900	~23

Important: the smaller the MDE and the lower the baseline conversion rate, the longer the test.
Asking to detect a +1% lift on a 2% CR means a test running into hundreds of thousands of users.

The three inputs that determine sample size

1. Baseline conversion rate. The lower it is, the more observations a reliable measurement needs.
A 0.5% CR requires four times the sample of a 2% CR.

2. Minimum detectable effect (MDE). What is the smallest lift that has practical value for the
business? An MDE of 5% needs four times the data of an MDE of 10%. The business sets the MDE from its
own economics, not from statistics.

3. Error levels α and β. The standard is α = 0.05 (a 5% risk of a false positive) and power
1 − β = 80%. For critical decisions, such as changing the homepage, raising power to 90% is sensible.

Common mistakes

Calculating the sample after the fact. Once the test has produced a significant result, a
retrospective calculation adds nothing. The calculation is a statement of intent made before launch.
Ignoring the number of variations. In an A/B/C test you have to correct for multiple
comparisons, or the combined type I error rate grows from 5% to roughly 14%.
Confusing absolute and relative MDE. A lift of 0.2 pp on a 2% CR is a 10% relative lift, not
0.2%. The difference in the resulting sample is fourfold.
Forgetting two-sided testing. If the test asks better or worse, z_α/2 = 1.96; if it asks only
better, z_α = 1.645. Using a one-sided test without an explicit justification is methodologically risky.

Frequently asked questions

How do you calculate the sample size you need?

For a two-proportion test the formula is n = 2 × (z_α + z_β)² × p̄(1−p̄) / (p_B − p_A)², where p̄ is the pooled conversion rate and p_B − p_A is the minimum detectable effect. In practice an online calculator is easier — Evan Miller's A/B test calculator, for example. Always set the requirement before launch, never after.

What happens if the sample comes out smaller than needed?

The test loses power, so the probability of missing a real effect — a type II error — goes up. The test ends without a significant result even though the improvement genuinely exists. On top of that, small samples make random fluctuations more likely to produce false-positive signals.

Does the number of variations affect sample size?

Yes. With three or more variations (A/B/n) the total sample grows in proportion to the number of comparisons against the control. You have to apply a correction for multiple comparisons — Bonferroni or an equivalent — or the combined type I error rate will exceed α.

Can a test be stopped early if the result is obviously significant?

In the classic frequentist approach, no: that is peeking, and it inflates the type I error rate. The alternatives are a Bayesian test, which stops on the probability that a variation is best, or sequential testing (e-values, SPRT) — both are designed for exactly this kind of continuous monitoring.

Gravity Field

Sample Size in Gravity Field

Gravity Field estimates test duration before you launch, from current traffic, the baseline conversion rate and the MDE you set. The test interface then tracks whether enough data has accumulated to decide.

✦Automatic calculation of the required sample before a test starts

✦Data sufficiency indicator inside the test interface

✦Low-power warning when traffic is too thin for the target effect

Explore the platform →