CRO is a discipline, not a list of tactics

Conversations about conversion optimization usually collapse into a list of tactics: strip fields
out of the form, add a countdown, change the button colour. The tactics exist, but they are not CRO —
they are hypotheses, each of which holds true in only part of the possible contexts.

CRO is the process that answers two questions: which changes to test first, and how to tell a real
effect from a random fluctuation. Value does not accumulate in a single winning test. It accumulates
in a body of knowledge about what works on one specific audience.

The difference shows up over a year. A team without a process has shipped twenty best practices and
has no idea which of them helped and which did damage. A team with a process has run twenty tests,
won five, lost seven and landed eight neutral — and knows exactly which five changes stayed in
production and how much they bring in.

The CRO loop

1. Data           → where and how much is lost (absolute numbers, not percentages)
2. Hypothesis     → “Change X for audience Y will lift metric Z, because…”
3. Prioritization → ICE or PIE: what returns most effect per unit of effort
4. Experiment     → A/B test with a calculated sample and a fixed end date
5. Decision       → roll out / roll back / reformulate
6. Knowledge base → the conclusion is recorded whatever the outcome

Step six is the one teams skip most often, and it is the step that turns a pile of tests into a
discipline. A losing test is not wasted traffic, it is knowledge: this audience does not respond to
that lever, so the next twenty hypotheses can be built on a different plane.

Hypothesis format. A hypothesis without a rationale cannot be validated. A workable statement
holds four elements: what changes, for whom, which metric it is expected to move, and what
observation the expectation rests on.

Moving the delivery-time block above the price on the product page will raise add-to-cart
conversion for mobile traffic, because session recordings show that users scroll down to the
delivery block before deciding, and delivery time is the number one question in support tickets.

Where hypotheses come from

Source What it gives Limitation
Funnel analysis A quantitative picture of losses step by step Shows where, not why
Session recordings Observed behaviour of specific users Easy to over-read a single case
Heatmaps How attention and clicks are distributed Merges different intents into one picture
On-site surveys Reasons for dropping off, in the user’s own words The people who answer are not the typical visitors
Support tickets Real barriers that escalated into a complaint Skewed toward already-engaged users
Internal site search Demand the site does not satisfy Needs query cleaning first
Competitor analysis Ideas and reference points Their solutions were never tested on your audience

Quantitative and qualitative sources work as a pair. The funnel answers where to look; qualitative
data suggests what to assume. A hypothesis built on session recordings alone risks fixing a problem
that affects a fraction of a percent of traffic.

Prioritization: ICE and PIE

Framework Axes When it fits better
ICE Impact, Confidence, Ease Small team, fast iterations
PIE Potential, Importance, Ease Many pages with very different traffic

Each axis is scored from 1 to 10 and the result is the average. The accuracy of those scores is
secondary: the point of the procedure is not the number but the argument it forces. When one person
puts Impact at 9 and another at 3, it turns out they understood the hypothesis differently.

💡

Add a traffic filter on top of the score. A hypothesis on a page that sees a few thousand visits a
month physically cannot produce a statistically significant result in reasonable time at a realistic
effect size. Changes like that are either shipped without a test as obvious improvements, or shelved.

Mistakes that void the result

Mistake How it shows up What to do
Peeking The test is stopped the first time the numbers look good Fix the end date and the sample size in advance
Insufficient sample The test “showed no difference” on knowingly thin traffic Calculate the MDE before the start; no power means no test
Optimizing a micro-conversion Button clicks went up, revenue did not move Keep revenue per visitor as the primary metric
Many changes in one variant The variant won, but nobody knows on account of what Split into separate tests if traffic allows
Local maximum A run of small improvements has hit a ceiling Test radical variants from time to time
Ignoring segments The average result comes out neutral Read the breakdowns, but as a source of hypotheses, not as a conclusion
No record keeping Six months on, nobody remembers what was tested Keep a test register with hypothesis, metric and outcome

Two of these deserve a longer note.

Micro-conversions. A micro-conversion is a useful diagnostic signal and a dangerous target metric.
An aggressive pop-up lifts subscriptions and lowers revenue; a removed filter speeds up the path to
the product page and raises returns. The primary metric has to sit as close to money as possible, with
micro-conversions kept as supporting numbers.

Segment breakdowns. Check a result across twenty segments and one or two will show a “significant”
difference purely by chance. An effect found in a breakdown is a hypothesis for the next test, not a
decision ready to ship. It is confirmed formally only by a separate experiment designed for that segment.

CRO and personalization

CRO looks for the single variant that works better for the whole audience. That is the right starting
position, but it has a ceiling: if one segment wins with variant A and another with variant B, the
combined test shows a draw and both potential gains are lost.

Approach Logic What gets rolled out
Classic CRO Find the best variant for everyone One winning variant
Personalization A different variant for different segments A rule that selects the variant

The practical takeaway: a neutral test result does not always mean the idea does not work. It is worth
checking whether new and returning visitors, mobile and desktop traffic, or paid and organic sources
diverge. If the divergence is stable and explainable, that is a case for personalization — one that
still has to be confirmed by a separate experiment.

Experiment launch checklist

  1. The hypothesis is written down in the format “change → audience → metric → rationale”.
  2. There is exactly one primary metric, as close to revenue as possible; secondary metrics are fixed in advance to watch for side effects.
  3. MDE and sample size are calculated, and the expected duration follows from them.
  4. The end date is fixed before the start and does not move during the test.
  5. Traffic splitting is verified — the actual group shares match the configured ones.
  6. The test covers whole weekly cycles, not just weekdays.
  7. The result is read as uplift with a confidence interval, not as a point estimate.
  8. The conclusion goes into the register whatever the outcome — losing and neutral tests included.
  9. A holdout group is left in place for long-lived changes — it answers whether the effect still holds a quarter later.