A/A Testing

A/A Test · Platform Validation Test

Definition

An A/A test is run before the first A/B experiment, or after technical changes to the analytics stack, to confirm that traffic allocation, conversion tracking and the statistics engine behave correctly.

Why an A/A test exists

An A/A test is a technical health check of the platform before any experiment. Both groups receive
an identical version of the page. If the system is sound, the difference in metrics should be
statistically insignificant.

Without it, a team risks collecting false positives and false negatives in its A/B tests — and
making product decisions on them.

What an A/A test verifies

Randomisation — whether traffic is genuinely split evenly between groups
Metric stability — whether conversion rate, average order value or revenue per visitor
fluctuate abnormally with nothing changed
Attribution — whether conversions are credited correctly inside the attribution window
Sample ratio mismatch (SRM) — whether the actual group split matches the expected one

Reading the result

Result	Interpretation	Action
p > 0.05, groups close	The platform is behaving correctly	A/B tests can start
p < 0.05, significant difference	A fault in the system	Inspect randomisation and tracking
SRM detected	Filtering or blocking is distorting the sample	Fix the split conditions

Usual causes of a failed A/A test

Visitors jump between groups on each visit because sticky assignment is missing
Bots or internal traffic are not filtered out
The conversion event is tracked twice
The A/B platform’s JavaScript tag initialises late, producing flicker

Frequently asked questions

When should an A/A test be run?

At the first launch of a testing platform, after technical changes to the tracking stack, whenever A/B results look suspicious, and when the analytics or attribution tooling changes.

What does a significant A/A result mean?

If an A/A test shows a statistically significant difference, the testing platform is broken. The usual causes are faulty randomisation, an error in conversion tracking, or a sample ratio mismatch where the actual group split differs from the expected one.

How long should an A/A test run?

Usually one to two weeks, long enough to accumulate a representative sample. The exact duration depends on traffic: at 10,000 unique visitors a day, 7–10 days is normally enough for a reliable check.

Gravity Field

A/A Testing in Gravity Field

Before the first A/B test, Gravity Field recommends an A/A run to verify that the integration and the tracking are correct.

✦A built-in sample ratio mismatch (SRM) check in every test

✦Integration diagnostics before the first experiment

✦Real-time monitoring of metric stability between groups

Explore the platform →