How to design an A/B test whose results you can trust
A/B testing is a powerful instrument — but only when the experiment is set up correctly. Mistakes made at the design stage produce unreliable results and push the business in the wrong direction. Here is how to plan an experiment so that the conclusions are accurate and actually useful.
1. Define the goal of the test
A goal is not simply “improve conversion”. You need to state clearly which business metric the test touches and which hypothesis it verifies. Examples:
Fashion: “If we increase the size of the images on the product page, average order value will grow.”
Electronics: “If we add video reviews, conversion to purchase will increase.”
2. The difference between a hypothesis and a fact
It is important to understand that any idea about a possible improvement stays a hypothesis until it has been tested. A common mistake is treating assumptions as facts — and that leads to the wrong decisions.
For example, if you assume that “bigger discounts increase sales” but never test the hypothesis, you can end up in a situation where profit falls because margin has been eroded.
3. Test one variable at a time
Change only one element at a time, otherwise you will not know what actually drove the result.
❌ Bad: you changed the price, the product card layout and the “Buy” button copy all at once.
✅ Good: you tested only the new product card design.
4. Calculate the sample size for the experiment
To get statistically significant results, the sample has to be calculated properly. An example:
- Current site conversion rate: 5%.
- Expected uplift: +10% (to 5.5%).
- Confidence level: 95%.
- Required sample size: roughly 15,000 users in each group (A and B).
With a smaller sample you risk results that are random rather than conclusive.
5. Rule out external factors
Seasonality, promotions, holidays and other factors can all distort the outcome of a test.
E-grocery: if you are testing a new category order in the catalogue, do not launch it right before the winter holidays — demand will spike anyway.
Furniture: a new discount is worth testing in a normal period, not in the middle of the sale season.
6. Split the traffic correctly
Groups must be random and equal in size. A classic mistake is sending only mobile or only desktop traffic into the test.
Example: in eCommerce, testing a new UX only on iOS and not on Android can produce misleading results, because user behaviour differs between the platforms.
7. Read the result correctly
“Conversion grew by 5%” does not automatically mean the test was a success. What matters is:
- Net profit.
- Average order value.
- Long-term indicators (repeat orders, returns).
A fashion case: the “Remove from cart” button was taken away — conversion went up, but so did the return rate. Net effect: a loss.
8. Analyse test results by segment
Checking the overall result matters, but analysing it across user segments matters even more. The aggregate picture can hide meaningful differences in how groups behave. For example:
- New vs returning users: newcomers may react to interface changes very differently, while loyal customers are already used to the old design.
- Geography: users in large cities and in smaller towns can have different behaviour patterns. In e-grocery, changes to the checkout flow may have a different impact depending on delivery availability.
- Device type: desktop and mobile users can perceive a page redesign or new CTA buttons differently.
Example: a test of a new product page UX may show an overall conversion uplift of 3%. But look closer and it turns out that:
- New users started buying less often (conversion −5%), because they do not immediately understand the new interface.
- Regular customers started buying more often (+7%), because choosing a product became easier.
Conclusion: instead of rolling the new design out to everyone straight away, run additional test iterations for the segments where conversion dropped.
9. Questions to answer before you launch
When designing the experiment, answer these questions:
- Which hypotheses are you testing?
- Which product changes are under test?
- Which user segments take part in the test?
- Which metrics will define success?
- What sample size do you need for statistically significant results?
- What will you do under each possible outcome?
Takeaway
An A/B test is not just launching two variants and picking the better one. You have to think the test design through in advance, account for the details, measure the result properly and avoid jumping to conclusions. Segment-level analysis helps you understand the impact of a change more deeply and adapt the strategy for different user groups. Follow these steps and you will make informed decisions that genuinely improve the business.
