Ways to Minimize the Time to Reach Statistical Significance

The time taken by the test results to achieve statistical significance depends on several factors such as:

• The current conversion rate of your website,
• The improvement you expect a variation to have,
• Number of visitors that become a part of the test each day after qualifying for conditions such as URL and segment targeting and %traffic included in the test,
• Number of variations tested,
• Traffic split between variations in the test, and
• Statistical significance threshold.

By controlling the above factors, you can influence the amount of time your test will take to get statistically significant results. Based on the amount of time needed to conclude your test, you can decide whether running an A/B test is the best approach with the scenario at hand.

If duration estimation suggests that your test could take a very long time to reach significance, consider one the of following ways to reduce the total amount of time:

1. Test Changes That Have a Significant Impact on the Conversion Rate Detecting a more significant change requires less traffic than detecting a small change. That’s because if the conversion rate of two variations is almost the same, there can be an overlap between the conversion rate ranges of the two variations. To figure out which variation is better, we need to reduce the overlap, and more data is required in the test to do that.
2. Increase the Coverage of the Test If you are testing something, say product pages of a category on your e-commerce website, you can consider testing all product pages and increase the number of pages on which this test is running. This will help you include more visitors in the test. You could also consider removing any visitor segment this test targets or increasing the %traffic included in the test if you have included less than 100% of traffic on the page.
3. Test Fewer Variations Suppose it is not possible to run tests that have a big impact on the conversion rate, and you are unable to increase the coverage of the test. In that case, you can consider reducing the number of variations in the test if possible. This can significantly reduce the amount of time required to reach significance.
4. Ensure Traffic Split is Equal If the traffic split between variations is not equal, consider changing it back to equal. With less traffic going to some variation(s) and more to other(s), the variation with less traffic takes longer to reach a confident conversion rate; hence, your test result is delayed.
5. Consider Using a Lower Statistical Significance Threshold If you are not testing something that can impact the bottom line too much, like changing the color of a button, consider using a lower significance threshold.

If your website has low traffic, and you are constantly struggling with testing on it, please refer to this blog which shares more details specific to what you can do in this case.