Sounds like more split testing is needed! Maybe site visitors are not clicking as often due to the color, not the words. Therefore, we should definitely drop option A from our Hello Bar message, but we need more data to conclude whether option C is statistically better than option B.ĭoes that mean the message seems to make a difference? Maybe, because if you were paying attention, we also had different colors for each message, so we have confounded our results with color. What it also tells us is that option B is not statistically better than option A, nor is it statistically worse than option C.
The upper right section tells us that the statistical difference occurs between A and C. If you start in the upper left, the p-value is 0.031, which is less than 0.05, so we conclude that at least one option is statistically different than one of the other options.
#Chi square test minitab free#
The top bar in green is a service we used called Hello Bar, which is free for use (one website only). We recently performed a split test on our website, using 3 different messages at the top of our website. Therefore, you either need to run a 2-sample proportions test on each comparison (A vs B, B vs C, and A vs C), or you can try Minitab’s Chi-Square % Defective analysis. Since the primary metric in split testing is a click rate (proportion), analyzing the split test cannot be done with an Analysis of Variance (ANOVA). In addition, what if we want to run an A/B/C test (3 options instead of only 2)? However, there may be situations where the percentages will be much closer together. In our example, assuming I had more than 100 site visitors, it is pretty obvious that the difference in the percentage is most likely statistically significant (not due to random chance). The click rate that is higher would be selected (“Free Sign Up”), and the other option is dropped (or modified again to run another A/B test).Įssentially, split (A/B) testing is a simplified hypothesis test or design of experiments (DOE). In the image above, you can see that 78% of the visitors clicked on the button when it said “Free Sign Up,” but only 34% of the visitors clicked on “Register.” The click rate is defined as the percentage of site visitors that clicked on the button divided by the total number of visitors who saw that option. After enough visitors arrive on the site (sample size is adequate), a comparison of the click rate is reviewed to determine which one did the best.