How the novelty effect arises
A user sees an interface change — a new recommendation block, a reworked product card, a different
CTA format. Curiosity about “something new” sharpens attention and can temporarily raise clicks or
conversion — not because the change is better, but because it is unfamiliar.
After a few visits users adapt. The effect disappears and the metrics fall back to baseline or below
it. Stop the test during the novelty peak and the team will roll out a “winner” that delivers no
long-term improvement at all.
What the novelty effect looks like on a chart
Conversion rate of variation B by day:
Days 1–3: 3.8% ← novelty peak
Days 4–7: 3.1% ← decline
Days 8–14: 2.7% ← plateau (the real level)
Control (throughout): 2.5%
Verdict: the real lift is +8%, not the +52% of the first days
How to diagnose it
Time segmentation. Compare week one against weeks two and three. A sharp drop in B’s advantage
is the signature of novelty.
New vs returning. New users have never seen the old interface — for them there is nothing “new”,
only a first encounter. Compare their behaviour with returning users: if the effect shows up only
among returning users, it is novelty.
| Segment | CR of variation B | CR of control | Lift |
|---|---|---|---|
| New users | 2.6% | 2.5% | +4% |
| Returning (week 1) | 3.9% | 2.5% | +56% |
| Returning (week 3) | 2.7% | 2.5% | +8% |
Important: a large lift among returning users in week one next to a small one among new users
is the classic signature of the novelty effect. The real lift is the one shown by “returning, week 3”
and by new users.
Common mistakes
- Peek-and-stop on day three to five. The test reads “significant” and the team calls it. Half of
that significance is a novelty artefact. - A single business cycle. One week is half a full purchase-behaviour cycle — weekdays plus a
weekend. You need at least two. - Large changes without allowing for the disruption effect. A full page redesign will depress the
metrics first; do not read that as “the change is bad” in the opening days.