A/B testing: when you really should
A/B is a refinement tool, not an exploration tool. When used correctly, it adds value; when used incorrectly, it wastes effort without yielding insights.
A/B is only worthwhile when the page has sufficient traffic, typically requiring thousands of interactions for the results to be statistically meaningful, and your question is which option to choose between two versions that differ by only one point. If the page has few visitors, observing five real users will provide you with answers much faster and at a lower cost. When the issue is still at a large structural level or you do not understand why customers stumble, qualitative research should be conducted first, not A/B.
- high-traffic page with the question of which option to choose between two different versions
- refining details that have worked like button text, titles, block layout
- With low traffic, observe 5 real users instead of waiting for enough data.
- The issue is still at a large structural level or not understanding why clients stumble.
A/B is a refinement tool, not an exploration tool. Sinh Vũ sees many business owners hear about A/B and think it is a scientific way to solve all design questions. In reality, A/B only answers a very narrow question: among two versions that differ by one point, which one leads to a higher conversion rate. That question is only valuable when you have enough data to listen.
Traffic is a prerequisite
To yield reliable A/B results, you need enough interactions to achieve statistical significance, meaning the certainty that the results are not due to chance. According to Nielsen Norman Group, this number is typically in the thousands and depends on the current baseline conversion rate and the minimum difference you wish to detect.
A newly launched page or a page with low traffic does not meet this condition. Running A/B tests in that case is not saving time, but wasting it waiting for data that will never be sufficient, then making decisions based on results that mean nothing.
When to and not to choose A/B
You should use A/B testing when: The page has sufficient traffic. The question is which option to choose between two versions that differ by only one point, such as the text on the call-to-action button, the title of a block, or the order of two content sections. You want to fine-tune details that have already worked, not redesign from scratch.
Avoid using A/B testing when: Traffic is still low; observing five to ten real users will yield results much faster and richer in information. The issue is still at a larger structural level, meaning you do not yet understand why customers stumble or which flow is appropriate. In this situation, qualitative research is needed first.
A/B only indicates which option wins, without explaining why. If you need to know why customers did not fill out the form or click the button, A/B cannot answer that.
Nielsen Norman Group, A/B Testing 101
The one element at a time principle.
A/B testing should only change one factor at a time. This is not a formal rule but a condition for the results to be usable. If you change the button color, headline text, and block layout all at once and one version wins, you won't know which factor made the difference. The results won't help you learn anything for the next adjustment.
The same applies to the duration of the run. It should run for at least one to two weeks to capture daily behavioral fluctuations throughout the week, and do not stop early just because one option is leading after a few initial days.
Common errors
- Running A/B tests on a page with few visitors and concluding without sufficient statistical significance.
- Changing multiple factors in one approach makes it impossible to determine the cause.
- Ceasing experiments as soon as a preliminary version is seen, without enough samples and duration.
- Use A/B testing to understand why customers stumble, while A/B testing only indicates which option wins.
- Ignore behavioral fluctuations by day of the week; running too short requires noise.
The viewpoint of Sinh Vũ
Sinh Vũ is a studio that designs and delivers websites. Most clients come to Sinh Vũ at the launch stage or have insufficient traffic for A/B testing to be valuable. Therefore, Sinh Vũ prioritizes establishing the right flow from the beginning through research and qualitative testing, rather than waiting for enough data to conduct tests.
A/B is a refinement tool for later stages, suitable when the page has sufficient traffic and your team is continuously optimizing operations. This is the phase after handover, and Sinh Vũ is ready to advise you when you reach that stage. However, if the page just launched today or customer traffic is still uneven, do not wait for A/B. Sit down with five real users and observe; you will learn much more in a few hours than in weeks waiting for data.
Decision checklist
Topic: When should A/B testing be conducted? Sinh Vũ guide, sinhvu.com
Select each item you find appropriate, then print or save as PDF to take with you.
If you have marked most of the signs above, this is the time to discuss in more detail. Sinh Vũ can help you review and propose a direction.
References
A/B Testing 101, Nielsen Norman Group. Measuring a 1% Increase in Sales Through A/B Testing, Nielsen Norman Group. How to calculate sample size of A/B tests, Optimizely.
Frequently asked questions
If the page does not have many customers, can it still run A/B tests?
You should not. With low traffic, A/B testing will not provide enough data to draw statistically significant conclusions, and you may prematurely stop when one option appears to lead, resulting in poor decision-making. Instead, observe five to ten real users directly: it's faster, cheaper, and will inform you why customers behave that way, not just which option wins.
Can I change the button color, title, and layout all at once to see which version performs better?
Do not do this in a single A/B test. When you change many things at once, even if one option wins, you won’t know which of those made the difference, and the results won’t be usable for the next refinement. A/B testing should change only one factor at a time to clearly identify the cause.
How long should you run A/B tests?
At a minimum, one to two weeks, regardless of how the results look on the third or fourth day. Customer behavior varies by day of the week; running too short will capture noise rather than real trends. The specific duration also depends on the current conversion rate and the level of difference you want to detect, so consider the sample size before starting.