In this piece, by Dan Olsen, you’ll learn about using analytics to optimize your product, by breaking down the lean product process into some concrete steps using real-life examples.
Meet Dan Olsen
Dan Olsen is an entrepreneur, consultant, and Lean product expert. At Olsen Solutions, he works with companies to build great products and strong product teams, often as interim VP of Product. His clients include Facebook, Box, Microsoft, Medallia, and One Medical Group. Prior to consulting, Dan worked at Intuit, where he led the Quicken product team.
He also led product management at social networking pioneer Friendster and was the co-founder and CEO of TechCrunch award winner YourVersion, a personalized news startup. Dan wrote the bestseller The Lean Product Playbook, published by Wiley, and organizes the Lean Product & Lean UX Silicon Valley Meetup.
The PM Motto
Today I’m going to be talking about how to use analytics to optimize your product. Let’s assume we’ve launched our product and now we have a product in the market, we can use analytics to improve it and optimize it.
So, I want to share a secret. It’s one of the most closely guarded PM secrets out there. And it’s the PM motto. So the product manager motto is similar to Spider Man’s but it’s a little different. It’s ‘with great responsibility comes no power’.
PMs are responsible for a lot of things. I had a lot of great discussions with PMs about all the things they need to do. If you don’t have a designer, then you’ve got to do some wireframes. You always fill the gap. So we’re responsible for a lot of things.
I’ve been speaking to audiences in product management since 2007, answering questions and building content and that’s what led to The Lean Product Playbook, which came out almost three years ago now. I wrote it as a comprehensive guide to all the things that you need to know as a product manager. And it’s meant to be a playbook to give you a hands-on guide with a six-step process.
Lean Product Analytics Process
The context for what I’m talking about today is: you’ve got your product in the market and you want to use analytics to improve it. Before you’ve launched your product, it’s hard to have any analytics. Once you’ve launched it, you can. And just like I had the lean product process for the pre-launch, I have a process for how to think about an approach using analytics to improve your product post-launch.
It all starts with figuring out what are the important metrics that we want to track, and the baseline value (where are we at today). It’s about figuring out for each metric, how much upside potential is it or what the ROI is if we try to improve that metric. This is all at the global level for your whole product and your whole business. And so what you want to do is look across those and figure out which one has the biggest potential.
Once you identify that metric, then you’re going to select that metric and focus on working on it. And then you go into a metric-specific optimization loop now for it. What we’re going to do there is basically as a team, we say we want to improve this conversion rate. We’re going to brainstorm different ideas for how to improve it and figure out which idea we think has the highest ROI. We’re going to design and roll it out. And we’re going to see how the metric changes. And we hope that it goes up to the right.
But if it doesn’t, we’ve still learned. So there’s value in going through the loop even if the first or second time you don’t move the needle a whole lot on that metric. We’re learning and iterating. And over time you will find something that moves that metric over time. You’ll find something else. Eventually, over time, it’ll start to get harder and harder. As you find the easy ways to improve that metric, it’s going to get harder and harder. You’re going to get diminishing returns.
So at some point, it makes sense to get out of this loop, go back to the global set of metrics and say “we fixed that conversion rate as much as it makes sense to, let’s go back and figure out what’s the next metric that we should be improving”. And when you’re thinking about metrics, there can be a million different metrics that you can track. It can be a little overwhelming, so it can be helpful to have a holistic framework for how to think about the metrics.
Dave McClure’s Framework: AARRR
A while ago, Dave McClure created a great holistic framework called Startup Metric for Pirates. So it’s a great framework because it applies to any business. It’s a way to just start with metrics at a high level and cover the landscape. And it’s AARRR because that’s the first letter of each of the five parts.
So the first one is acquisition. The acquisition is all about how to get people to learn about our website or our app. So if they don’t even know about it, how do we get them to the point where we can tell them about our app and hopefully convert them to customers. They’re not customers yet, they’re just prospects right now.
The next A is activation. And that means how do we convert those prospects? If we got them to our landing page or our site, what percentage of those are we converting to actual customers? How are we finding customers? It may be paying, it may be registered, or an email, whatever it is. I would usually call this conversion, but then it wouldn’t spell AARRR, like a pirate. So it’s called activation.
Then once they’re customers, by definition, they’re using your product. That’s what a customer means. They start using your product, not all of them keep using your product. Hopefully, they keep using it. That’s what the third letter, R, is which is retention. What percentage of people are continuing to stay active?
Then, the next R is referral, which is hopefully a function of people using your product, they like it. And they tell their friends. Josh Elman talked about three different types of reality. One is word of mouth. One is demonstration. The other one is you may have some social mechanisms where people can invite their friends. Or if you’re a communication product like Slack, you’re going to have some implicit viral loops there. So for B2B, this is often less important. So that one is the one that may not apply as much in your case. Certainly, for games and social products, it’s critical, but these other three are important.
And the final R is revenue. This is, by virtue of the customers using our product, we should be making some money. Either they’re paying us or we’re making money by monetizing their behavior. So that’s the last one. And the goal here is to, figure out which one should we be focusing on?
Check this out: Team Metrics for Product Managers
MTMM: Metric That Matters Most
There are five choices here, which one at a macro level offers the highest ROI? And so you want to focus on the right metric at the right time which I call the metric that matters most for right now. And that’ll change over time. MTMM for short: Right now for our product, or our business, what’s the metric that matters the most?
To do a quick little mini sample, let’s say we have a new product, we’re about to launch it next week. Or we just launched it say today. And we’re just trying to decide, should we focus on acquisition? Should we focus on getting more prospects to our webpage? Should we focus on conversion? You know, getting people that come to our webpage to convert to customers, or should we focus on retention? One of these is going to be better and offer the highest ROI right now.
There’s no one answer for every situation. But in general, the answer that I give is if we just launched our product, more than likely our retention is probably near we want it to be. Not where we want it to be. So we can go spend a lot of money on acquisition and buy Google ads and Facebook ads, but then people are going to come. We haven’t optimized conversion yet. So not that many people are going to convert. Our product isn’t too sticky yet so a lot of people are going to drop out.
So in general it makes sense to focus on retention first until it gets to a decent point. I talk a lot about the leaky bucket metaphor. Retention is basically how much water is staying in your bucket versus leaking out. Customers, being the water. Then conversion and then acquisition. That’s the way to get the most ROI on your marketing dollars in general.
There are a lot of different metrics you could focus on. In the context of product-market fit, which is what my book is all about, I think there’s is one metric which is retention rate. I just got done talking about the leaky bucket stuff. So I want to explain what retention rate is because there’s a certain way of looking at it, but it’s basically what percentage of people are remaining active over time. It’s best understood and usually displayed as a retention curve.
Let’s take an easy example. Say we just launched a product. We take all the customers that sign up in the first 30 days and we throw them in a bucket and we watch them. Not calendar days, but for each user let’s index the timeframe for when they joined and first started using it, and then let’s see how they drop off over time because they will drop off over time.
So the first thing you would notice on this curve is that it doesn’t start at a hundred percent. It starts at 27%. What does that mean? That means over 80% of people never came back after they use it the first time. It’s pretty brutal. There will always be a gap. Everybody loses people. They go and they either forget about it or they just realize it doesn’t meet their needs. There’s a lot of reasons why people don’t come back. So that’s just the reality. Then it decays and then it eventually goes down.
It can do one of two things at the end year. It can flatten out, which is the good case. But for most new products, frankly, it goes down to zero. It’s just a question of when does that happen? 30 days, 60 days, 90 days? What that means is that all those customers you work so hard to get in the bucket, eventually all leak out. If conversely, you’re holding on to some people that’s great. That means these people have stuck with you for 90 days. Your product is doing something valuable to them.
Now over time, as you improve your product, you’re going to want to not throw all the customers into one big bucket and analyze them as one big group. You want to slice them and analyze them separately and say “those were the guys in the first month when we had all those bugs and those issues. But we fixed those. Let’s look at the next group of people.”. That’s where cohort analysis comes in. All that means is dividing your user base into different segments and looking at the retention curves for each of those segments. Typically done by time, but it doesn’t have to be by that. It could be by acquisition channel or by customer segment.
Here’s an example of cohort curves. So we have three cohorts here. Again, we have what percentage of users are returning over time. This time we have weeks and signup instead of days, but we’ve got three cohorts. Cohort A is in the blue diamonds, cohort B is in the red square cohort C is in the green triangles. Imagine you were the product manager for this product, which cohort would you rather have as your cohort of users? Why would everyone pick the green cohort?
You might be interested in: These Are the Metrics Great Product Managers Track
Your brain knows what matters the most is what’s highest at the end. So the secret is if you want to have a quantitative measure of product-market fit that’s behavioral and not attitudinal, it’s where this line ends up. So what I’m telling you is if this flattens out, if it goes to zero, you don’t have product-market fit. If it flattens out the percentage of flattens out at that’s a measure of your product-market fit. It’s a real live believable measure. The higher it is the better.
Say we launch our MVP beta two years ago, 24 months ago. And this is our retention curve. We managed to hold on to say 5% of people after 12 months. Say in the next six months, we work hard listening to customer feedback, fixing bugs, adding features, fixing UI, fixing messaging.
What we hope to see is if we look at the next group of people that sign up six months later, that the whole curve moved up. And most importantly, that the tail value went up. Hopefully, in the next six months, we continue to better understand our customers and improve things. So what you want to see is where you’re at on retention and make sure you’re not at zero. If you are, get it flat and then steadily over time, improve it. That’s how you improve product-market fit and use analytics.
You might be interested in: Basic Metrics for Each Organization in your SaaS Company
Real World Data – Android App Store
I like to share real-world data to support my points whenever I can. This is real-world data from the Android app store. So these are retention curves, percentage of users still active. Days since app install. And what they’ve done is they’ve done four cohorts. They’ve done it in a very particular way. The red curve is the top 10 apps in the Android app store. The next curve is the next 50 or 100. The next is the next 5,000. So what does it show? It shows that the top apps have the best terminal retention rate and then the next best apps have the next best. So it just shows you that basically, it’s a direct measure of that.
The Equation of your Business
So you want to make sure your bucket’s not leaky enough. And then you can focus on conversion and acquisition, the other parts of the AARRR startup metric for pirates framework. At some point, you need to go beyond that framework. It’s great to get your head around the different five different areas and to focus and get retention down and get acquisition and conversion.
At some point, when you get to the revenue part, you need to take into account the specifics of your business because different businesses earn different revenue in different ways. That’s where the concept that I talk about in my book, the equation of your business comes in. I have a technical background engineering background. So if I want to optimize something, I want to be able to express it quantitatively as an equation.
And if you’re thinking that you don’t know how to approach your business as an equation, you can always start with profit equals revenue minus costs. And for most tech businesses, what matters the most is the revenue. The marginal costs aren’t as important as marginal revenue. And so the key here is to break the revenue down. And this is an action. When we say we want to grow revenue, that’s not actionable. What we want to do is break it down mathematically so we get to the point of having actual metrics.
I’m going to do that for a subscription business model. Let’s say we have a SaaS subscription product and we have a 30-day free trial that people can do. And then they have to upgrade. This is for a given time-period, a given month, a given year, whatever it is. We can break revenue down into how many paying users did we have x what was the average revenue per paying user? You may have heard of ARPU is an acronym for that. So it’s paying users x ARPU. That’s a way to break it down. Those two together, just equal that. And then we can break down the paying users into for this month that we’re looking at, how many new paying users did we get? And then how many repeating paying users do we have from the previous time period.
When you’re doing these exercises, almost always distinguish between new customers or users and returning basically. The repeat paying users just ends up being how many did we have in the last time period x 1- some cancellation rate or churn rate.
Back on the new users, again, I mentioned we had some free trial, so we can say, well, how many trial users do we have? And what was the conversion rate? That will give us how many new paying users that we have? And for the trial users who may have various marketing channels, like SEO, SEM, viral, and we have a conversion rate. So we started with the high-level revenue that’s not actionable, but now we’ve broken it down into several items that could be actionable. We could say: do we think it’s better to focus on the trial conversion rate, or is decreasing this cancellation rate the best way, or is it increasing the revenue per paying user, or is it fixing our market?
That’s the kind of discussion that you can have. And for SaaS products, because of this one minus factor, the churn rate can be very powerful. It may sound like a small improvement to take your churn rate from 6% to 3%. But it has a non-linear improvement in like revenue and lifetime value. So anyway, what we want to do is break this down for your business.
Check this out: What To Do With a High Churn Rate
View Each Metric as a Gauge
What I think about each one of these detailed metrics, I view it as a gauge or a dial. And a gauge or a dial is a numerical measure. It has a minimum possible value. It has a maximum possible value. This would be for a percentage metric. It goes from zero to a hundred and then it has where’s it at today.
So that’s a current value. But the way I think about each metric is what’s the minimum value, the maximum value and where are we at today? And that’s the framework to figure out which ones are the critical few. As we look at each of those gauges, how much upside potential do we think there is for each of those? How much do we think we can move the needle?
If we redo the design of a page, for example, we can move the conversion rate from 5 to 10%, what would the revenue impact be by using the equation of the business? We can calculate what that would be and how many resources do we think that it would take to move the needle that much. It’s usually developer time, maybe money. And then what we want to do is look across the metrics and find which one we think offers the best ROI.
Types of Metric ROI Situations
There are typically three ROI metrics or ROI profiles.
1. Good ROI
The way I show this is we’ve got return and investment. The yellow dot is where we are today and the white line shows if we pursued the best ideas and we invest this much, how much return will we get?
2. Bad ROI
We’re higher up on the curve and so we can put in a lot of investment, but it’s not going to lead to a lot of return. And those are the main ones that you see.
3. Great ROI
There’s a third type that you can see if you analyze things carefully. If you analyze things just right, you realize that there’s something that with a little bit of investment can make a really big improvement in the return. So the return curve is fundamentally different shapes. I call these silver bullet opportunities. And they do exist, especially if you haven’t started doing analytics optimization.
Friendster Case Study
I want to close out with a case study of applying the lean analytics process from end to end. Let’s pretend you’re the product manager here. I used to be the head of product manager of Friendster back in the day. It was the first social network. And so it was a great opportunity for me to work on viral loop optimization back in 2004.
When I showed up at Friendster, everyone in the building was saying viral growth is critical. We need to get new users from our existing users without paying. It was implicit. Everybody believed that. And I’m like, great. What have we done on that? Nothing. I think I said “okay, I’m going to apply my process. And I need to figure out what framework and how to approach this.”
So the first thing I did is define the viral loop and then figure out what metrics I can put on there. So the viral loop is basically, we have active users that are using our site. They have the opportunity to invite their friends. Not all of them do, but some of them do. The friends get an email saying ‘Dan Olsen wants you to join Friendster. They either click on the email or they don’t. If they click on the email, they get to the registration process. And then if they get through it, they become users. Some of which remain active. So the first thing I did is I define this UX flow and where people could drop off. The next thing I did is figure out what metrics I can instrument here to track and optimize this.
The first metric was the percentage of active users. Not all of them are active. The next thing was the percentage of users that are sending invites. Not all of them are sending invites in a given month or week or whatever. There’s a second factor. When you invited your friends, it’s not like you had to invite one at a time. You can invite multiple. So there were invites per sender as well to capture that step. The next one was just the click-through rate on those invitation emails. And the next one was the registration conversion rate. What percentage of people are getting through?
So those were the five metrics that I use to instrument that loop. And now multiplied together, they determine your viral ratio. And now I had to decide which one I should focus on. I’m the PM. I’m the head of PM for Friendster, I’ve defined five metrics. Which one should I focus on? I want you to picture that you’re in those shoes.
Let’s make it simple. Let’s boil it down to these three: percentage of users sending invites, invites per sender, and conversion rate. Which one would you focus on again? The popular answer would be the conversion rate. The next step in the process is what are your baseline values? I didn’t show them to you previously but here are the baseline values. The percentage of users sending invites was 15% of users. Average invites per sender on average, when people invited friends, was 2.3 and the registration conversion rate was 85%. Now that we have baseline values, why don’t we re-think the question? Now, more people would like to focus on improving the percentage of users sending invites. See that’s the power of baseline. You have no idea how invites work or anything. You just know where you’re at. And now fewer people would focus on conversion rate.
So what’s going on? There’s a quick hack. Your brain knows this stuff just by knowing the baseline value. It’s called the upside potential of a metric. That’s what I call it. And not knowing anything about the metric or the product, you can usually make a pretty good guess on which one is going to offer the best upside potential.
Remember I said, each of these is a gauge and has a minimum value. What’s the minimum value for registration process yield? Zero. Maximum value? A hundred because it’s a percentage. What’s the baseline value? Where are we at today? We’re at 85%. So what we can do is quantify the max possible improvement is. I’m not saying we would get it. But if we just had the most amazing outcomes and ideal results, what’s the most we could do? We’re at 85%. So the max is 15 more points. If we divide that from where we are, it’s an 18% improvement.
The second one, the percentage of users sending invites, also goes from zero to a hundred percent. We’re at 15%. How much headroom do we have here? We have 85 percentage points that we can go up. We divide that by 15 that’s 570. That’s why your brain just knows. 15 versus 85, it’s a no-brainer that it’s got a bigger upside potential.
This third one, the average number of invites per sender, what’s the minimum value for that one? Zero. What’s the maximum value? Infinity. We’re at 2.3. Question mark divided by 2.3 equals what? I don’t know, but there’s a really good chance. It could be even bigger than this one. It’s fundamentally a different metric. That’s why it’s important to think about the min and max. These are percentage metrics.
Improving The Metric
Are you feeling a sense of Deja Vu from something I said earlier? I said that there are three profiles: a bad profile of ROI metric. That’s what that the registration process yield is. We’re at 85%. It’s going to take a lot of effort to not get a lot of return. The second one is a good ROI curve which is the percentage of users sending invitations. We feel like there’s a lot of upsides that we can do. The last one, we’re not sure yet, but it seems like this average number of invites per sender could be a silver bullet metric because it’s got near-infinite potential right up there. So for those reasons, we decided to focus on this.
The next would be doing the process. We’re going to focus on an average number of invites per sender as a team, let’s brainstorm all the different ways we can move that metric. How do we increase the average number of invites being sent out per sender?
And for each idea, we try to estimate the expected benefit and how much it will improve the metric. How much do we think it will move that needle on if people are inviting 2.3 friends today, will it move it to 3, 4, 5, 6? What do we think it will do? And then we think about the expected cost. How many engineer weeks or days is it going to take? And you want to run some high-level ROI filters and figure out what you want to do.
Back then, one thing you didn’t know is to invite friends, you had to mainly type in their email addresses. There wasn’t an address book importer. And so we were the first social network to add that. We decided let’s add it. And then again, we had to do the ROI because you had to build one each for Gmail, Yahoo, Hotmail. We realized that most of our users are on Yahoo, so we decided to do an MVP of that and see what it does.
So we’re cruising at our baseline value which is 2.3. We roll out the new feature and what we saw is more than double. It took a little while because of the 7-day average, but it more than doubled. So it turned out to be a silver bullet feature. Because we doubled that, we doubled the number of active users, new users, and revenue that we had. So I was pretty excited about that result. It only took one engineer one week to build it. So high ROI and then we replicated it for Hotmail and Gmail and the other ones. And we saw an even bigger increase.
Read next: Optimize Your Product Management Analytics and Metrics by Dan Olsen
So to sum up the lean product process in the following steps:
- Start out by figuring out what your metrics are
- Measure the metrics baseline values
- Evaluate which one has that upside potential.
- Select top metric using the upside potential hack to figure that
- Get it into the metric loop
- Brainstorm ideas to improve the metric
- Identify the highest ROI idea
- Design and implement it
- Analyze how the metric changes