How to Drive Product Growth with A/B Testing

In this talk, Mark Marcello explains several key topics in Product Management, including why growth experimentation is important, what things a company should do to make money, best practices, interactive case studies, and the power of intuition.

Meet Mark Marcello

Mark Marcello

Mark Marcello is a Product Manager in CROmetrics where he is responsible to facilitate A/B Testing for the company’s clients, including Airbnb and Viking Cruises by leveraging the tool Optimizely.

He is an e-commerce product management expert who holds over a decade of progressive professional experience and has demonstrated the ability to combine strategy, creativity, and strong business acumen with well-developed technical talents that position companies for success. 

How A/B Testing Can Unlock the Full Potential of a Growth Strategy

We’ll start off by asking ourselves, what exactly is A/B testing? The easiest answer is actually the Google definition:

“A/B Testing is a randomized experiment with two variants, A and B. It includes the application of statistical hypotheses testing as used in the field of statistics.”

But that this doesn’t tell us, is that it’s not as simple as A/B, although that’s what we call it. Many tests involve multi-variant testing, (A and B / A, B and C / A, B, C, and D).

A/B testing can be as simple as tweaking the taxonomy of a CTA, or as complex as overhauling the whole page itself.

‘Do the Thing That Makes the Money’

Ultimately, testing tells you more about your user experience and customer journey, and ultimately you want that data to tell you what your customers want. Having that information will drive your product forward, so you want to make testing and experimentation a part of your core business strategy.

Testing will help you to identify and focus on the singular, most important metric. This is the metric which will help you to learn the most about your users. (It’s not the only one that matters, you’ll want to set up multiple secondaries to help fill out the bigger picture.

Consistent Growth is a Continuous Cycle of Experimentation

In Sales a common expression is ‘always be closing.’ Well as a product growth professional, you want to always be testing.

  • ABT (Always Be Testing)
    • Ideate: Bring strong ideas to the clients. Look at your webiste/app and think ‘where I can optimize? Where can I hack growth?’
    • Plan: Strategise how to go forth with the idea. Create a roadmap and prioritization goals.
    • Build: Develop. Input client feedback
    • Launch
    • Analyse: Assess how a product can be made better. Time to iterate!

In case you were wondering how the experts do it, Amazon views experimentation as critical to growth:

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.

Jeff Bezos

While Facebook’s goal is to fail, and learn from that failure…

The strategy of Facebook is to learn as quickly as possible what our community wants us to do – and that requires a culture that encourages people to try things and test things and fail, and it also requires building infrastructure to enable people to do that.

Mark Zuckerberg

Why Don’t Companies ‘Do the Thing That Makes the Money’?

Many companies aren’t embracing experimentation as quickly as competitors, for a number of reasons.

  • Testing and research has historically been difficult
    • Expensive
    • Time consuming
    • Unreliable
    • Competitively complicated
  • Experimentation program success requires speed and infrastructure (however, this can now be outsourced)
  • Cultural blockers can encumber testing strategy

“Old” Approach to Satisfying Customers: Not Data Driven

A lot of companies believe in one way of doing business, despite the current landscape of their industry evolving beyond it. (Think of Bed Bath and Beyond still sending cut-out coupons in the mail.)

  • Prior experience and company “Folklore”
  • “This is how it’s always been done”
  • “Mailing 20% off coupons is what works”

Many businesses are still relying on a very antiquated way of making decisions, a way that is not data driven. Board members who hold a lot of authority, but not a lot of in-depth customer knowledge, say ‘hey, that looks good. Let’s do it!

The Creative Function Should Be Data-Driven

Creative direction is not exempt, and should also be data-driven.

  • “Brand guidelines”
  • “Creative know-how”
  • Leverage creative talent informed by customer data

Testing as a Strategy: Real Estate Pop Quiz

Can Intuition and Experience beat data-driven experimentation? Take a look at this client’s app, and notice the four icons which have been highlighted:

A/B Testing graph variation A

Now look at the variable for the test. Very little has changed, except for a label added to each icon. Can your intuition and experience tell you which one was more popular with users?

A/B Testing Graph variation B


Variation B wins because it’s far easier for the customer to figure out what each option does, when it’s clearly labeled, as opposed to when it’s just represented through icons. If you guessed right, you proved the value of intuition, and that it should never be ignored even in this data-driven age.

How to Develop Strategic Customer Insights that Drives Growth

1. Heatmaps Show Us Where People Actually Click: Click and scroll data shows you exactly where users are going on your site, and is a great visual tool. You can see here that the main CTA is very ‘hot’ but the third icon from the left is actually also an important CTA. In tests, you could run an experiment where less emphasis is put on other icons to draw more attention to this one.

2. Analytics Offer Insights on User Behavior: Tools like Google Analytics are accessible to everyone, and can help to drive you in the right direction.

3. Customer Feedback Yields Winning Copy Ideas: Talking to users helps to get you more descriptive terminology. It’s quite literally the voice of the customer, which can drive both testing and marketing.

4. Tailored Experiences Based on Inbound Traffic Source Drive Increased Efficiency Normal Site PPC Landing Page: Tailoring where you send your users to based on where they’ve come to you from can make a huge difference to the user experience. See the example below:

PPC vs Normal Site graph

5. Experiment Results Yield New Ideas to Test: Sometimes data defies intuition. In the example below, the option on the left looks much cleaner and less clunk than the variation on the right. However, the right variation one, because it helped to make this clearer for the user.

Experiment results yield new ideas to test

Kickstarting a Testing Strategy That Creates a Culture of Experimentation

Identify the One Thing That Makes the Money

You want to identify what your key KPI is, as other companies have done:

  • Facebook: Daily Active Users
  • Uber/Lyft: Weekly rides
  • Optimizely: Monthly experiment starts
  • Airbnb: Nights booked
  • CROmetrics: Weekly experiments run

Prioritize Tests to Maximize ROI

The RICE method is one of the three most popular prioritization methods which helps to make product decisions. This can help you to decide which tests to run.

RICE scoring prioritization method

Help to work how how high a priority something is by measuring its potential Reach, Impact, Confidence, and then divide the sum of those scores by Effort. To see more on how this is done, check out 3 Prioritization Methods All Product Managers Should Know.

Testing as a Strategy: Final Exam 

One final exam, to see how your product intuition measures up against a data-driven test.

Graph Pay rate
Graph note from manager


Test results
Test results graph

Though the results for both the variations were similar for completed applications, the Note from Manager added a personal touch and hence led to an increased down-funnel metric. The note also mentions first come, first serve basis, which triggers a sense of urgency in the customers.

To Summarize…Do the Thing That Makes the Money!

  • Make testing and experimentation, a strategy
  • Develop strategic customer data
  • Kickstart your program with or without a partner
    • Identify one, most important, metric
    • Prioritize experiments
    • Accelerate throughput to increase ROI

Mark shows that one should never ignore their intuition and always go with the data. He also emphasizes that a long copy always isn’t necessarily a bad thing. It might prove to be more effective when used in the right way and has appropriate content.

Product Mindset CTA

Enjoyed the article? You may like this too: