A/B testing: when you really should
A/B is a refinement tool, not an exploration tool. When used correctly, it adds value; when used incorrectly, it wastes effort without yielding insights.
A/B testing is only worthwhile when the page has sufficient traffic. Typically, thousands of interactions are needed for the results to be statistically significant. At that point, your question is which option to choose between two versions that differ by only one point. If the page has few visitors, observing five real users will provide you with answers much faster and at a lower cost. When the issue is still at a large structural level or you do not understand why customers are stumbling, you need to conduct qualitative research first, rather than using A/B testing right away.
- high-traffic page with the question of which option to choose between two different versions
- refining details that have worked like button text, titles, block layout
- With low traffic, observe 5 real users instead of waiting for enough data.
- The issue is still at a large structural level or not understanding why clients stumble.
A/B testing is a refinement tool, not an exploration tool. Sinh Vũ sees many business owners hear about A/B testing and think it is a scientific way to solve all design questions. However, in reality, A/B testing only answers a very narrow question. That is, among two versions that differ by one point, which one leads to a higher conversion rate. This question is only valuable when you have enough data to analyze.
Traffic is a prerequisite
For A/B testing to yield reliable results, you need enough interactions to achieve statistical significance. According to the Nielsen Norman Group, this number is typically in the thousands. It depends on the current baseline conversion rate and the minimum difference you want to detect.
A newly launched page or a page with low traffic does not meet this condition. Running A/B testing in that case only wastes time waiting for data that will never be sufficient. Ultimately, you will make decisions based on results that do not mean anything.
When to and not to choose A/B
You should use A/B testing when: The page has sufficient traffic. The question is which option to choose between two versions that differ by one point, such as the text on a call-to-action button, the title of a block, or the order of two content sections. This is when you want to fine-tune details that have already worked, not redesign from scratch.
You should not use A/B testing when: Traffic is still low; observing five to ten real users will yield results faster and with much richer information. The issue is still at a large structural level, meaning you do not yet know why customers are stumbling or which flow is appropriate. In such situations, qualitative research is needed first.
A/B testing only indicates which option wins, not why. If you need to know why customers are not filling out a form or not clicking a button, A/B testing cannot answer that question.
Nielsen Norman Group, A/B Testing 101
The one element at a time principle.
A/B testing should only change one factor at a time. This is a condition for the results to be usable. If you change the button color, the title text, and the layout of the block all at once and one version wins, you will not know which factor created that difference. The results will not help you learn anything for the next refinement.
The same applies to the duration of the test. You should run it for at least one to two weeks to capture the behavioral fluctuations by day of the week. Don’t stop early just because you see one option leading after the first few days.
Common errors
- Running A/B tests on a page with few visitors and concluding without sufficient statistical significance.
- Changing multiple factors in one approach makes it impossible to determine the cause.
- Ceasing experiments as soon as a preliminary version is seen, without enough samples and duration.
- Use A/B testing to understand why customers stumble, while A/B testing only indicates which option wins.
- Ignore behavioral fluctuations by day of the week; running too short requires noise.
The viewpoint of Sinh Vũ
Sinh Vũ is a studio that designs and delivers websites. Most clients come to Sinh Vũ at the launch stage or have insufficient traffic for A/B testing to be valuable. Therefore, Sinh Vũ prioritizes establishing the right flow from the beginning through research and qualitative testing, rather than waiting for enough data to conduct tests.
A/B testing is a refinement tool for later stages, suitable when the page has sufficient traffic and your team is continuously optimizing. This is the phase after delivery, and Sinh Vũ is ready to consult when you reach that stage. But if the page just launched today or traffic is still inconsistent, don’t wait for A/B testing. Sit down with five real users and observe. You will learn much more in a few hours than in a few weeks waiting for data.
Decision checklist
Topic: When should A/B testing be conducted? Sinh Vũ guide, sinhvu.com
Tick each item that applies or that you can already answer, then print or save as PDF to take with you.
If you have marked most of the signs above, this is the time to discuss in more detail. Sinh Vũ can help you review and propose a direction.
References
A/B Testing 101, Nielsen Norman Group. Measuring a 1% Increase in Sales Through A/B Testing, Nielsen Norman Group. How to calculate sample size of A/B tests, Optimizely.
Frequently asked questions
If the page does not have many customers, can it still run A/B tests?
You should not. With low traffic, A/B testing will not have enough data to draw statistically significant conclusions. You may prematurely stop when you see one option leading and make the wrong decision. Instead, observe five to ten real users directly. This method is faster, cheaper, and will tell you why customers behave that way, not just which option wins.
Can I change the button color, title, and layout all at once to see which version performs better?
You should not do this in one A/B test. When you change many things at once, even if one option wins, you will not know which of those changes made a difference. The results will not be usable for the next refinement. A/B testing should only change one factor at a time to clearly attribute the cause.
How long should you run A/B tests?
At least one to two weeks, regardless of how the results look on the third or fourth day. Customer behavior varies by day of the week; running too short will capture noise rather than the true trend. The specific duration also depends on the current conversion rate and the level of difference you want to detect, so calculate the sample size before starting.
This article is for reference. The scope, pricing, and specific commitments of Sinh Vũ are detailed in the proposal and signed contract.