If you have built a website, AB testing is definitely one of the essential tools for you. It helps you test how the new website design or feature is going to impact your sales. To understand if a new feature is going to work or not, you have to enter the number of website visitors and the number of conversions in the AB testing significance calculator. Now, in this post, we are going to tell you what statistical significance is and how to test it using an AB testing calculator.
What is Statistical Significance?
When it comes to AB testing experiments, statistical significance is the method that helps you differentiate versions of your website’s features. There are two versions of your website features, one is the control version, and another is the test version. When you are using statistical significance, it will help you figure out which of these versions has the ability to improve your website’s performance. For example, you have created a new feature for your website and you are going to run a test with a 95% significance level. Now you can be assured that 95% differences between the control version and test versions are real.
Statistical significance is mostly used in websites or business websites to find out how the new features are affecting your business’s conversion rate. Statistical significance can be found on your website by surveying your visitors and existing users, and by examining the survey results, you can find out what your customers prefer. For example, if you are creating ad campaigns: campaign A and campaign B and you are surveying your users to understand how many people prefer campaign A and how many people prefer campaign B. But before choosing one option based on your users’ choice, you will have to find out which one of these options provides more statistical significance. To calculate statistical significance you will have to use a statistical significance calculator.
How to Calculate Statistical Significance?
The first step of calculating statistical significance involves creating a hypothesis. When you are testing new features of your website, you will have to create a null hypothesis and an alternative hypothesis. In the null hypothesis, you will find that there is no link between the features that you are comparing. And in the alternate hypothesis, you will find that there is a connection between the two versions of the features that you are testing with AB tests.
In the conversion rate AB testing, you will have to create a hypothesis that generally involves buttons, images, or webpages to find out if they are going to improve your website conversion rate over time. By using surveys, you can understand the point of view of your users about a particular feature. Surveys help you test for concepts like we talked about in the ad campaigns in the section above.
After creating null and alternative hypotheses, you can conduct AB tests to make sure that the hypotheses are valid. To understand the validity of your null hypothesis, you can refer to its Z score. The score can also inform you if there is no connection between the versions of the feature that you are comparing. In fact, it will also inform you if there is any strong proof between the null and alternative hypotheses.
In the statistical significance AB testing, you have to determine if your tests will be one-tailed or two-tailed. One-tailed tests come with an alternate hypothesis that will involve directional effects. The two-tailed hypothesis comes with the negative effects of the version of the feature as well as a directional effect. Most testers conduct the two-tailed test for a better understanding of the feature and its possible conversion rate.
But whether you are in it for two-tailed tests or one-tailed tests, you will have to use a statistical AB testing calculator. This calculator will help you calculate the significance of two versions of the same feature.
What Does an AB Test Calculator Do?
With the AB testing calculator, you must get the answers to your pre- and post-test analysis questions. The test calculator should answer the following questions:
- Is the test variant better than the original version?
- How long should you test for getting the conversion effects?
- What is the required measurement of the model for the test?
- What is the momentary ROI for the test variant?
- Does the test have a required duration?
With AB test significance, you can get conversion rate which is calculated by conversions/traffic*100= conversion rate. You can also calculate uplift which is the relative increase of the conversion rates measured between campaign A and campaign B. To find the negative uplift of your web page, you will have to apply this formula: conversion rate A/conversion rate B* 100= uplift. However, when you are using these formulas, you will have to remember that these are the increase in the rate of conversions and they don’t mean actual sales.
In the AB testing statistic, your results will be referred to as significant when testing is confident about the result. When you are achieving statistical significance with a 95% confidence level, it implies that your results are not accidental and will occur once in every 20 times. But to get AB test significant results, you will have to increase your sample size, create a larger uplift, or create more consistent data with fewer variants.
How to Increase Sample Size?
There are two steps to increase your sample size:
- If you run your test for a longer duration, the chance of risking your data will be higher. Most web browsers will delete cookies within a month or after two weeks and AB testing tools need those cookies to understand and divide your users into group A and group B. The longer you run the tests, the higher your risk will be of cross sampling.
- You can invert more of your website traffic to your test pages by connecting a link from your home page to your landing page. But if you try to interlink your web pages, your number of traffic will be based on different categories and the traffic might not be the same kind of people because they are searching for different things.
How to Increase Uplight?
The other three steps to create a larger uplift:
- You can create a more significant variation using button colors, CTA texts, CTA titles, etc. to create an impact on your visitors in some of the cases. But at different moments, you need to do substantial edits to create an effect. Your variations are going to create a better impact and uplift when the variation conveys your offer to your target audience.
- You can also create a variation with the version of your web page and persuasive notification. The notification feature will create an impact on your visitors.
- Another way to increase uplift is to reduce friction on your webpage by reducing the number of spaces you have on your contact forms and creating more text and visual graphics.
Conclusion
However, when you get significant results, you should not rush to edit your website because statistical significance does not provide any practical information. Sometimes AB tests can still give false-positive results, and in that case, you should change the website’s elements one by one and in a gradual manner. When you are optimizing a part of your website and not the other, it can create negative effects on the other parts and in that case, it can reduce your conversion rates. That’s why you should conduct AB testing using an AB testing calculator but your best shot will be to modify the things slowly and watch out for the KPIs on your website.
My name is Vijay Singh Khatri, and I enjoy meeting new people and finding ways to help them have an uplifting experience. I have had a variety of customer service opportunities, through which I was able to have fewer returned products and increased repeat customers when compared with co-workers.
