3-4-13. Case Study: A/B tests

When companies want to test new features or versions of a web page, they often use a method called A/B testing. The way this works is that one set of users called the control group is shown the old version of a page while another set of users called the experiment group is shown the … Read more

How do we make a recommendation when three out of our four metrics had a significant difference for testing each metric individually, but insignificant differences when we use the Bonferroni correction? The Bonferroni method is a conservative one, and since we expect the metrics to be correlated, this would be best handled by a more … Read more

As you’ve seen in previous lessons, the more things that you test, the more likely you are to observe significant differences just by chance. This happens when we run evaluations from multiple metrics at the same time. The probability of any false positive increases as you increase the number of metrics. Luckily, this is something … Read more

In addition to computing the enrollment rate, we can also compute the average reading durations with this dataset. The two analyses so far were comparing proportions. With this metric, we’ll be analyzing the difference in means. This analysis will be quite similar. Since we’re comparing reading durations, we only care about view action. So let’s … Read more

The second change Audacity wants to try is on their course overview page. They created a new description for one of their courses to dedicate larger portions to connecting concepts in the course to career skills, and less on the details of each concept. They hope that this change may encourage more users to complete … Read more

As you saw in the last section, this dataset includes view and click actions on the home page of Audacity’s site, from users that were shown the control and experimental versions of the A/B test. Our task is to analyze these actions to see if there was a significant difference in performance for the two … Read more

The first change Audacity wants to try is on their home page. They hope that this new, more engaging design will increase the number of users that explore their courses. That is, move on to the second stage of the funnel. Now that we know the change we want to make, we need to choose … Read more

In this case study, you’ll be analyzing AB test results for an online education company called Audacity, that offers courses on finance. Let’s first see what a typical user flow might look like on Audacity site. New users would land on the home page, and if they’re interested, they would click to explore courses. As … Read more

Great job completing this case study. To summarize, you learned about the uses and values of AB testing, defining metrics that measure changes in your experiments, analyzing results with confidence intervals and hypothesis testing, handling multiple metrics, and common difficulties associated with AB testing. Congratulations again on completing this case study.

Welcome to this case study on AB test. In this lesson, you all apply what you’ve learned in the previous lessons, to help a company decide whether to launch two new features on their web site. You’ll do this by analyzing results from a widely practiced experiment called AB test. The work you’ve done so … Read more