A product manager is a multifaceted professional who must be a natural problem solver. To face day to day challenges, there is a wide variety of tools to which s/he can turn to. Here we teach you how to use one of the most important product management skills: Product A/B testing.
What is Product A/B testing?
In simple terms, it is an experimental method that compares two variants – A and B – with the objective of determining which variants produces the best results. Also known as split testing, A/B tests consist of dividing an audience into two or more groups (depending on how many variants are being compared) and presenting each group with versions of the same impulse or content. The difference between the versions usually lies in one variable, which allows the tester to isolate the cause of the difference in results.
The image below shows an example of an A/B test ran by Amazon on the day that this post was written. The sole difference between the two versions of the homepage presented is the second image of the carousel.
What can be A/B tested?
Almost anything can be subjected to A/B testing. Websites, emails, ads, and even app interfaces are frequently tested. Besides, there are tons of different variables that can be manipulated to generate variants for the test. Some of the most common variables are:
- Call to action text
- Call to action color
- Call to action location
- Subject lines (for emails)
- Ad copy
Before you decide what to test, first formulate a hypothesis of how these variables can affect your results. This will allow you to make a decision of which variable should be tested first.
How long does a Product A/B test last?
The duration of the test can vary depending on how many versions are being tested at the same time. The amount of traffic you are receiving is also an important metric. You want to allow the test to run for enough time so that each version is viewed by enough people. This will enable you to extract unskewed conclusions. However, you should also avoid running the test for too long. Results could be influenced by too many outside impulses over which you have no control.
A/B testing can help a product manager to improve results, from top of the funnel (as click through rate or bounce rate) to lower funnel outcomes (such as revenues). At Product School, we teach aspiring product managers skills like A/B testing. We invite you to go to our website and check out the complete syllabus, to get a full view of our 8-week program.