Updated: May 6, 2024- 11 min read
As a Product Manager, you know metrics can help you make better product decisions. But not all metrics are made equal. While some metrics might look impressive, they don’t necessarily translate into business success. So how can you recognize vanity vs strategic metrics and ensure you’re focusing on data that gives meaningful results?
We’ll take a closer look at vanity vs strategic metrics and how to choose the right ones, but first of all, let’s do a quick recap. Why is it so important to get metrics right?
Vanity vs Strategic Metrics
Just like a diamond ring from Tiffany’s vs the knock-off from your local market, there’s a clear winner when it comes to vanity vs strategic metrics. While they may look the same, only one of them is worth the investment in the long term. So how can you tell the difference between vanity vs strategic metrics, and why is the former so maligned? Let’s start with vanity metrics.
Put simply, vanity metrics are any type of data that make your business look good but aren’t indicative of real performance. For example, let’s imagine you’re a product leader at a software company and you see the new product you’ve developed has over 1 million users. You may think, “Wow, that’s a lot of users, we must be doing something right.” But don’t get blinded by the dazzling lights of that vanity metric. Digging deeper into the numbers, you’d see that 50% of those users are inactive and 20% of those users have unsubscribed. Not looking so good now, is it?
Aside from lacking context, vanity metrics are often uncontrollable, making it difficult to repeat your results. For example, imagine to your huge surprise that Elon Musk praises your new social networking app on Twitter. You see a marked increase in your product’s app store views. This may lead to a huge boost in app downloads, but is this success repeatable? If you can’t increase app views again in the same way, this is a clear sign you’re looking at a vanity metric.
What’s more, unlike strategic metrics, vanity metrics do not provide you with actionable information about performance. For example, it may be a great ego boost to see your company has millions of Instagram followers, but does that information give you any useful business insights?
But perhaps the main reason to be wary of vanity metrics is that they don’t necessarily translate into sales or help you to achieve other important business goals. Your meditation app got 2 million downloads. That’s great! But did the users actually subscribe to your company’s premium service that drives your revenue?
Don’t get us wrong, just like admiring yourself in the mirror when you feel your new outfit is on point(!), there’s nothing wrong with paying attention to and taking pride in vanity metrics, but they shouldn’t be the sole basis of your product strategy.
What are some examples of vanity metrics?
Wouldn’t it be simple if you had a set list of vanity metrics that you would know to avoid? The truth is, there’s no such list because what constitutes a vanity metric is very subjective. In fact, any metric from engagement to revenue can be a vanity metric. It’s all about the context and how you’re using the data to inform your product strategy. What could be considered a vanity metric in one setting may be a strategic metric in another. But if you’re still craving a list, here are some telltale signs you have a vanity metric on your hands.
Characteristics of Vanity Metrics:
overly simplified
lacking nuance and context
misleading
can be easily gamed and artificially increased
can’t be sustainably reproduced
don’t provide you with actionable business insights
don’t contribute to your business objectives in any meaningful way
If your metric ticks one or more of these boxes, chances are it’s a vanity metric.
8 Characteristics of Strategic Metrics
Now that we know how to detect a vanity metric, let’s take a closer look at eight characteristics of strategic metrics and how they can help you make better product decisions. Strategic metrics should be:
1. Tangible
Good metrics should be tangible. That means they should represent a clear, understandable goal that everyone on the team can 'feel' and connect with. Examples could include the total number of people using a product, the total number of items sold, or the total number of minutes spent using an app or service.
2. Backed up with counter-metrics
Secondly, good metrics should have visible counter-metrics. As we saw earlier, only tracking one or a very limited number of metrics can make you lose track of other equally important information. For example, if we only measure customer acquisition without measuring customer satisfaction, we may find that we gain a lot of new customers in the short term, but fail to retain them in the long term. You can avoid this by tracking counter-metrics as diligently as primary metrics.
3. Controllable
Thirdly, metrics should be controllable in order to be meaningful and boost morale. This means that teams should have the ability to influence their metrics based on their actions. For example, using average customer wait time as a KPI at a taxi call center is a metric that is not controllable. There are many factors that could cause the wait time to increase, from rainy weather to large events – things an employee has no control over but which could directly affect their performance rating. Aside from sinking morale, uncontrollable metrics don’t add a lot of value. A more controllable and useful metric in this situation could be the number of resolved customer support tickets or quantitative customer satisfaction feedback.
4. Fast
Strategic metrics should also be fast. This is because as a Product Manager, you need to be able to receive feedback quickly after making changes to know if you're moving in the right direction and change course if not. This fast feedback loop is vital to facilitate learning and iteration.
5. Testable
As well as being fast, strategic metrics should be testable. A/B testing in particular is a powerful tool to learn and improve quickly. Teams that use testable metrics can easily test out hypotheses, pinpoint what’s working and what’s not, and identify future areas for improvement.
6. Explainable
Another hallmark of strategic metrics is that they’re explainable. If a metric changes, we should be able to understand and explain why it changed quickly, or at least come up with a plausible hypothesis. This includes decomposing the metric across various dimensions, like location or customer segment, to understand the drivers of change. After all, if you see significant changes to a metric, but can't explain the reasons behind it, is it really offering you meaningful insights?
7. Hard to Game
No cheating! Good metrics should be hard to game. Metrics can create powerful incentives, so we need to consider how companies and individuals may twist the data in their favor. For example, if teachers’ performance bonuses are based on their students’ grades, you can see how they may be tempted to grade papers a little too generously. It’s important to be on the lookout for such unintended consequences and put systems in place to prevent gaming, for example, tracking counter-metrics as mentioned earlier.
8. Rarely Averages or Percentages
Lastly, good metrics are rarely averages or percentages because they often fail to provide a comprehensive and meaningful understanding of the underlying data or performance being measured. One reason for this is that averages or percentages alone may lack the necessary context to interpret the data accurately, making it challenging to identify outliers, trends, or patterns that could be crucial for understanding the underlying factors influencing the metric.
What’s more, averages can be heavily influenced by outliers or extreme values, skewing the interpretation of the data. For example, if the average response time of a system is calculated, a few unusually long response times could significantly inflate the average, making it an unreliable measure of typical performance.
Instead of relying solely on averages or percentages, it’s often beneficial to use absolute counts as well as distribution measures, ratios, trends, and qualitative feedback. These additional metrics can provide a more comprehensive understanding of the data, uncover patterns, identify outliers, and facilitate deeper analysis and decision-making.
Metrics in Action: A Practical Example
Now that you know the differences between vanity vs strategic metrics, it’s time to see them in action. Let’s look at a practical example.
Imagine you’re the new product manager for a pet-sitting app. Your team tells you they weren’t too sure which data to track so they’re tracking almost all metrics. You decided to investigate and identify the most relevant metrics.
You open your metrics dashboard and see the following information: Total Pet Owners Registered, App Downloads, Booking Conversion Rate, and Pet Owner Reviews and Ratings. Let’s look at them one by one and decide if they’re strategic metrics that offer real value, or whether they’re simply vanity metrics.
1. Total Pet Owners Registered: You feel a surge of pride as you open your metrics dashboard and see a huge number of pet owners registered on your platform. But wait a second. Does this metric paint the whole picture? Does it help you to understand how many pet owners are actively using the app and finding suitable matches for their furry friends? Or does it mask a significant percentage of users who registered but never actually use the app.
Verdict: Without more context, this is just a vanity metric.
2. App Downloads: Next you move on to app downloads. Your app download count seems healthy and has risen since last month. Time to pop open the champagne? Not so fast! You dig a little deeper into the numbers and find that of those users only 40% are active and the percentage of active users has been gradually decreasing month by month. Yikes! It would have been very misleading to base success on the number of app downloads alone.
Verdict: Without further data, such as the number of active app users, the number of app downloads could be a misleading vanity metric.
3. Booking Conversion Rate: We’re not having much luck so far. Let’s see if booking conversion rate proves any better. To your delight you find you have a high booking conversion rate, indicating that your app provides a seamless journey for pet owners, from searching for sitters to confirming bookings and resulting in satisfied customers and increased revenue. Does this metric give you useful insights that help you make informed product decisions? Yes!
Verdict: Booking conversion rate is a valuable strategic metric that helps you to understand the effectiveness of our app's booking process and the overall user experience. This tangible metric clearly feeds into important objectives such as increased revenue.
4. Pet Owner Reviews and Ratings: This next one leaves you with stars in your eyes! 1 million 5-star reviews! That means 90% of all reviews have been rated as excellent by pet owners. But have you just been dazzled by another vanity metric? Let’s find out. Does this metric offer you actionable insights to inform your product review? In this case, yes! User-generated reviews and ratings are precious gems for your pet-sitting app! They offer valuable insights into pet owners' experiences and the quality of service pet sitters provide. This helps you to find out what’s working and what’s not as well as building trust and credibility, attracting even more users to your app.
Verdict: Pet Owner Reviews and Ratings is a fast, tangible, and hard-to-game metric that gives you a treasure trove of insights into customer satisfaction, helping you to make the constant iterations that are necessary to become the world’s leading pet-sitting app.
Congratulations! Now that you’ve identified the metrics that truly matter, you can keep them in focus, perhaps with the help of a metrics tree. You can use these strategic metrics to inform product decisions that will contribute to the overall success of your platform.
Guiding Principles for Choosing Effective Metrics
As we said earlier, there’s nothing wrong with feeling the warm glow of pride that accompanies vanity metrics, as long you don’t base your product decisions on them. And there are no hard and fast rules about what is considered a vanity metric. In one situation it could be seen as a vanity metric, while in another scenario it could be a strategic metric. It all depends on the context. The main thing to consider when deciding to track a metric is whether or not it’s an accurate indicator of performance and allows you to glean actionable insights that can help you achieve your key objectives.
Discover More with Product School
Found this post useful? Take your product management skills to the next level and learn how to use data to generate expert-level insights with our Product Leader Certification.
Enroll now and learn directly from proven Product Leaders at top Silicon Valley tech companies like Google, Meta, and Amazon, leveraging their real-world experiences, frameworks, and approaches to help you excel.
Updated: May 6, 2024
