How to Use Data to Drive Product Decisions by PayPal Senior PM

This month, we’re focusing on all things Product Data Analytics. Keep an eye out for events, podcasts, blogs, and more!

Data analytics is one of the most important tools in growth, so it’s essential for Product Managers to understand what kind of data they need to collect and why. Luckily, Deb Dutta, Senior Product Manager at PayPal, has a lot of experience in the area and talks us through her data process from discovery to revenue, as well as sharing her 5 techniques for making data-based product decisions.

About Deb

Deb brings over a decade of experience in Product & engineering leadership roles. She is a Computer Science major with a Master’s from Carnegie Mellon University. Deb has worked in tech since the beginning of her career in big companies such as IBM, Oracle, Microsoft, and Cisco.

Today, she leads PayPal’s AI& ML platform enabling targeted marketing and personalized experiences to PayPal’s customers. She is also the chapter lead & founding member of Women in Product SF Peninsula region. She is a strong advocate for women in tech and mentors women across the bay area as part of several non-profit organizations.

How to Use Data to Drive Product Decisions

Today, we’ll start by talking a little bit about my background so there’s context to some of the stories I tell. We’ll talk about what Product Managers do, and how we use data to make intelligent product decisions. And then finally, we’ll talk about using talk about how to data and product thinking in Product Management interviews—or if you’re looking to hire Product Managers, these are some of the structures you can look through when you’re hiring your candidates!

So a little bit about me. Like I mentioned, I’m a Senior Product Manager at PayPal. I manage our real-time enterprise decision-making platform. So we essentially provide risk and compliance decision capabilities to all of our PayPal customers. This is based on data science and machine learning models. I moved to the states about 11 years ago, I’m from India and I studied engineering as so many of us do coming from India.

I came to Carnegie Mellon to do my Master’s in Computer Science, and then I started my first job at Microsoft as a developer. Over the last 10 years, I’ve worked in multiple domains starting with databases, middleware, networkings—which is hardcore CC ++ coding in transport layer. I worked on 4G LT mobile gateway protocols. Eventually, I moved into storage with Hitachi systems, where I started my first Product Management gig.

And that was where I got a real taste of incubating a small startup into a large company and really taking a product from concept to revenue. We grew our team from 5 to 30 engineers, and were working on providing a visualization platform for police officers to do real-time decisioning based on things that are happening around them.

So when they get a 911 call, what information can you arm your first responder with so that he actually makes a good decision? That is, is there bloodshed, were there witnesses, did someone tweet about it? Was there a partial license plate catch? Is it a terrorist?

Police officers on a scene in NYC

This is called situational awareness. And I’ll walk through some of the examples about how we went about creating that product considering I don’t know anything about policing. The one good thing about it was it was also a brand-new initiative for Hitachi. It was their first stake in the ground as a storage company for their IOT initiative in the space of video intelligence and public safety.

A lot of learnings and a lot of mistakes along the way, but it was all in good spirit.

Aside from that, I also moonlight as a fashion/travel lifestyle blog. I have about 170,000 followers on Instagram. The reason I bring this up is because a lot of my growth has been purely based on data. I use data and analytics to constantly monitor where I’m getting engagement from, what customers are actually reciprocating with my posts, what time zones, what locations, what type of content resonates with what kind of folks. I’ll use some of those examples to walk us through how, if you have a platform, a blog, or just a product like Facebook and so on, how do you actually monitor your success based on some of your engagement metrics?

Product Starts With People

You might work in a team that is very engineering-driven, with a lot of your roadmap and features driven by an engineering-specific mindset. Or you might work on a team where a lot of your decisions are purely driven by a Product Manager. I personally have worked on both kinds of teams, and I consider the two of these as extremes. I’ll tell you why.

There’s this whole notion about the Product Manager as the CEO of Product. I don’t really love that statement because as a Product Manager, you have all the responsibility of a CEO and none of the authority of a CEO. It’s a really ambitious statement to say the Product Manager is the CEO of Product, so let’s do a reality check.

Realistically, what you have as a Product Manager is you have a goal: build your product. You need to come up with a plan to drive your entire team along to actually meet that goal. You have some success criteria that you are evaluated on, that your product is evaluated on, and your team is evaluated on. And the sad part about it is a lot of times, Product Managers have little to very little information about what you’re going to encounter along the way. You can have changing timelines, you can have dependencies go awry. Maybe the hypothesis that you began with doesn’t work out.

How do you ensure that you are set up for success so that these obstacles don’t come in the way of your team moving forward? For me, the whole journey of people-process-product starts with people. I emphasize paying attention to people that you recruit along with you on this journey.

Think about an uphill climb to top of Yosemite. You really don’t know what obstacles you’re going to encounter along the way. However, if you recruit the right people who give you different perspectives, they’re the ones who are going to keep your feet on the ground and get you moving forward.

In our case, you think about engineering, of course. They are your resource. But what about UX? What about someone who’s giving you that perspective and keeping you real about what really matters to the end-user? What about architecture?

You do this to make sure that your implementations are not hacks, they are actually are industry standard. They are scalable, your product is robust.? So you need someone who is giving you the right advice from an architectural perspective.

And then the last point of view you need is project management. They keep you grounded when it comes to your sprints, your dependencies, your commitments, and your timelines.

two women in an office setting looking at a laptop together

That’s why the people that you surround yourself with are so important. As a Product Manager, your first goal is actually empowering your team: creating a safe environment so that every single one of these people who has a role in your team has the ability to express themselves, so you’re not making decisions alone.

Read next: Leading Good Product Teams

Now that you’ve set out a plan and you start down that journey, and you measure your progress based on the KPIs you set out, every single one of these people is going to help you make a sound decision as you collectively move forward. You’re going to meet obstacles, obviously, but you recalibrate your plan and you move forward as you go along until you hit the end of the line.

Product Discovery: Get into Your Customer’s Head

You might have heard of The Lean Startup by Eric Ries. If you haven’t read it I recommend you do. It’s really a Bible for any of us who are on the Product Management journey. It’s not the only Bible, but it’s Bible enough for me. What The Lean Startup really advocates for is the Build, Measure, Learn method, which is inspired by Toyota’s lean manufacturing methodology.

And a huge takeaway from this is the customer first method. The idea behind this is: before you determine what to build, you need to know who you are building it for.

Discovery, as many teams call it, is where Sense and Seize comes into play. Go through a detailed Discovery process to really learn about who your customer is, what their problems are, and to seriously develop empathy for those problems. That way when you come up with solutions for your customers, you can really get into their heads to solve it.

Discovery can be of two types: quantitative and qualitative. I recommend both types. Quantitative is where you’re reaching out to the masses. You’re trying to get themes about what possible problems are out there, who the different customer segments are that are relating to your product in a positive way, and who is not.

And the way you’re going to get that data is to send out surveys, with questions that are pointed, but not leading. There are tons of survey tools out there that’ll give you very clear insights, like: women within the age group of so and so from this location respond positively to my product. There are ways to draw these clear insights, and that’ll give you a good starting point to get to the next phase of your discovery, which is qualitative.

In qualitative, you identify focus groups in areas where you think you need to focus and get to the next level of detail. That’s where interviews come into play. Interviews are very tricky. You don’t want to go with leading questions, like Do you not like the XYZ feature of my product? No. Instead ask, What do you think about the XYZ feature about my product? Non-Leading.

Learn more about customer interviews: Do’s & Don’ts of Interviewing Customers to Build Great Relationships

Customer-Facing Product Discovery: The Hitachi Visualization Suite

I’m going to give an example of what we did in my previous job. Like I mentioned, my product was a situational awareness tool for first responders. The way this plays out typically is we get a sales call. A salesperson gets us a request saying this is a feature you guys need to go off and build.

I’ve never ever ventured into the police force and I never will. Yet me and my developers are going to come up with a tool that’s going to help solve and fight crime. Sounds crazy, right?

The only way you can do this is by learning and talking to go through a day in a police officer’s life. This is what they walked us through, and we walked through different segments of police departments, some large, with larger budgets, dealing with high crime. Some of them are more tech-savvy. Some of them not so much, some of them were really remote, etc. They had a lot of differences.

police station sign

But one common pattern that seemed to arrive from all of them was I get a 911 call; someone says gunshot; I need to go and dispatch my paramedics and first responders. And when I arrive at the crime scene, how much information do I actually know? Was that a terrorist? Was it a hit and run? Was there someone who actually saw witness?

Hearing this, my team had a lot of ideas: we can partner with gunshot companies; we can partner with license plate reader companies, and so on.

But the police were like, Yeah, we already have all that. That’s not my problem. But my problem is when I get that call, I don’t have enough time to check everything separately. I’m going to log into one camera to see what’s going on there, then I’m going to log into some gunshot detection system to see where the gunshot came from, etc.

Instead, they wanted a single pane of glass, like on Google Maps. This is how they described it to us, “We want a Google Maps.” So they have a view of all the assets that they own: cameras, gunshot, sensors, license plate readers, facial recognition, pictures. When you click on a pin, instead of seeing a street view, you get to see a live view of what’s going on. And let’s say someone tweeted saying, “I saw a suspect. He bolted off in a car wearing a black hoodie. This is a license plate I was able to capture.” They wanted access to all of that.

So from here we had a starting point. There were a lot of random requests aside from this that some people wanted: “We want all of social media incorporated. We want drone detection software.” Those are one-offs, and they might be great. But what you’re trying to really arrive at is your MVP. MVP comes from, what do you hear that’s common across your customers that’ll make them buy your product? They have to pay money for you to actually get to the next level, past MVP.

Read next: The Difference: Prototype vs MVP

So we took what was common, came up with some mockups, and went back to these groups.

And it’s not just about coming up with one MVP, you might go through several iterations. When you come to the build phase of your product, you go through milestones. You come up with, something like clickable mockups, take it back to these user groups and see how they’re interacting with an applicable mockup. You haven’t even invested resources in it. Just see how they’re reacting to it.

And as you advance through these milestones, you validate with the KPIs that you set out. Am I getting closer? Am I getting closer to what this customer will pay for? That’s what the build phase is.

And then as I mentioned, once you roll your product out, you measure, with the goal of learning. Learn from the mistakes.

You start out with some hypotheses: I think that I will have 80% engagement in this police department, but I see 2%. Now you know it’s 2%, but how do you deep dive? How do you get to the next level? How many times are you going to go back to the customer and ask them, why didn’t you click on this? You can’t do that. You need data, instrumentation that will give you these clues as to where you need to begin your investigation. Data plays the most critical role in actually getting you to the next level.

You might also be interested in: Why Data Analytics Matters for Product Managers

Backend Discovery: PayPal’s Risk Platform

I’ve been talking about consumer-facing products, which I wanted to touch on because some of you may be in consumer-facing roles. A lot of times when we talk about Product Management, the examples are about consumer-facing products.

But there are so many of us in the industry that are actually on backend systems. We’re working on platform, and that’s a whole other beast. At PayPal, I’m part of the risk platform team. This is nuanced because I have to enable my internal customers, who are consuming APIs of my platform to generate value for the end-user. The difference is, my KPIs are very closely tied to my internal customer’s KPIs, like engagement rate for example.

What’s interesting about risk is we don’t manage revenue. We are the bad news folks, we manage losses. It’s a very interesting empathy-building dynamic, where risk Product Managers have to be a little bit more careful about so that we’re not letting our risk bias. We are the paranoid freaks; we’re like, No, you cannot do this because you could possibly be a fraudster. You’re trying to hack our system.

But I have to draw that balance between declines versus user experience. Every time you go to France or Italy and you try to swipe your ATM card, most times you get declined. That’s your fraud algorithms kicking in, and that’s great. You will never hear a story saying that PayPal got hacked and all of your financial information was leaked. Never.


And the reason behind that is our super-advanced machine learning algorithms. Our data scientists are working day and night just to make sure that that doesn’t happen to any of you folks. However, that’s not easy. Because as a risk Product Manager, I can’t decline you every time you log into PayPal because you’re going to stop using it. But if you get hacked, that’s it. You’re never going to come back. It’s a fine balance that you need to draw.

Read next: An Amazon PM on New Product Thinking in the Age of Machine Learning

I like this challenge. I know people think about Product Management as primarily with consumer-facing products. I have been on that side of things, but this is a whole other beast. So I want to throw this out there in case you guys are interested in something a bit more challenging.

That being said, my journey is the same. It’s just my customers are a little bit removed from the external customer. They’re internal customers who are developing APIs on top of my product, so I have to be conscious about making sure my product is generic.

A merchant team might bring me a requirement saying that “You guys really need to up your fraud detection algorithms, I’m seeing a lot of losses,” versus a consumer team or the Venmo team comes and tells me “Guys, you’re totally screwing my experience. I’m losing engagement, and the whole thing about Venmo is we are doubling our engagement every quarter.”

I need to make sure that the features are released, satisfy everybody. You have to consciously make that balance. And this is where people ask me, Are you a technical PM? Are you a non-technical PM? There is actually some value in thinking about that question, because when you are a technical PM, you have to make those sound judgments, right? You can’t just make decisions based on business goals or customer goals; there are also platform goals that we need to think about.

Defining KPIs

Now let’s get on to the crux of why we’re here today: data. What questions can data answer for you? This is really about figuring out what KPIs matter to you and how these KPIs help you make decisions.

Engagement and User Churn

When you’re talking about conversions, there are many methods to determine where you are going to lose customers. And how do you measure conversion? If you have end customers come to your product, how many of them bought the product? Or something as simple as: you sent out an email saying, “Hey, sign up for Facebook.” How many people actually went ahead, filled in their email address, filled in these X things, and hit sign up?

And as you go through these measurements, you want to be able to see out of, let’s say a hundred users, that 50% of them dropped out. Where did that 50% come from? You want to see where you’re losing people so you can start focusing on them. This funnel analysis will help you determine where the churn is happening, where you’re losing your users from the system, and what helps them convert across.

The second thing I want to talk about is user engagement. What feature is driving your engagement? When PayPal actually has its earnings call and it says “engagement has been better than ever.” And they throw out a percentage. But how did they get that number?

over the shoulder shot of someone on their phone

Companies like Facebook and LinkedIn do this, I do this for my blog all the time. I’m constantly measuring how many users are new versus how many of them are return users. How many of them are coming back to see my blog? Because retention is a very important metric.

In my previous product, the Hitachi Visualization Suite, we started noticing that people would log in for five minutes and log out, as though there was no crime happening in the city. But that’s not true.

How do I get to the bottom of that? Is it because cameras were not streaming? Is it that they were not even able to log in? Data was telling me that they were logging in, but sessions were really short. And like I said, retention is probably one of the biggest metrics any of us will think about. Building great products is great, right? Getting people to sign up for that first experience is wonderful. But either the experience was really terrible so people left, or something that you did basically did not delight the customer. So how do we measure retention? Things like the subscription renewals, return users.

Now you need to get to the bottom of why people leave. How do you do that? Think about the last thing that the user did before they churned out of your system.

In my case, I had these cameras that I put out on Google Maps where they click. And most times when my users are logging in it’s because there’s a crime that occurred; they’re clicking on something to immediately take action. Data can tell me that, How many times do users get a camera streaming error?

Now I know that the reason someone left my system is because my freaking product did not stream the camera. And that’s really bad for me because helping solve crime is my whole value prop. So measuring things like that will help you get to the next level.

Check out: How to Optimize Your Product Using Analytics

I want to talk about behavior, which comes a little bit more into play when you think about marketing. Let’s say PayPal introduces credit cards and you have multiple marketing campaigns, Facebook, Google Ads, email, etc. How do you know what’s really effective? Understanding what channels are driving engagement will tell you what your user behavior is. And then you can go ahead and dissect it.

User Segments

Another way to look at it is segments. With Hitachi, we targeted different segments with the different kinds of police stations. Are you thinking about the size of the police station, which comes with its own nuances? Or the location of the police station, which could mean bandwidth issues and servers being overloaded? You need to consider all of that.

Return On Investment

And lastly, the most important thing of all, is ROI for the investment you’re putting in. How quickly are you going to turn profitable? All of these metrics have to help you answer that one question, because that’s what your execs are looking at.

5 Techniques for Data-Based Product Decisions

Here are five simple techniques that have used in my process, and we’ll walk through each of these:

  1. Funnel analysis
  2. Cohort analysis
  3. Segmentation
  4. Event-based tracking
  5. Clickthrough rate

Funnel Analysis

Funnel analysis is really about determining churn at each stage. I’ll walk you through a simple example. This is most applicable when there’s a workflow that you’re analyzing, so let’s say you have a workflow, like “sign up for an account” or “download and install this app,” etc. That’s something that is a multi-step process. This will help you determine what percentage of users are completing what steps.

Let’s say you’re launching PayPal in a new country, hypothetically. You’ve run some marketing campaigns, and you’ve got a whole bunch of users that have decided to begin the signup process. Now, as you walk through, you put in your email address, you get an email into your email account saying, “click on this link to verify that this is your email.” Then you go back and now you have a profile, but to complete the profile you add a credit card or a bank account. Once you do that, there’s verification that happens. And now you have an active account.

More quality content: The Results are In: The Future of Insights is Bright

This is a multi-step process that a Product Manager who’s responsible for signups would be thinking about. Obviously not everybody who starts the journey completes it, but you can see the folks who you’ve lost along the way and can try to reduce that loss.

Now it’s time to think about hypotheses on why people drop off. Think about, why did people not make it to X stage? Is it because I’m letting you add a card, but I’m not letting you send money to your friends and family? In that case, maybe it’s because I’m offering limited capabilities. Could it be that there are competitors in that market that I haven’t thought about that are offering better capabilities? Could it be brand awareness, that nobody knows who the heck PayPal is? And so now you have some hypotheses that you can analyze to determine which one is true, so that you can take action on it.

Application: Deb’s Blog

I’m not allowed to share PayPal data, so I’m going to walk you through how we can use funnel analysis for my blog. I use Google Analytics heavily, which actually could be a good testing ground for you if you’re just getting into analytics. Create a simple website so you get comfortable with what and how these analytics work. Because when you go into your actual jobs, I’m sure your companies have proprietary tools that help you walk through all of this, but this gives you a hands-on experience of how you can use this data because it tells you very clearly what your drop off is and so on.

My Google Analytics shows me a lot of folks coming in and a segmentation based on country. A lot of people are coming to my landing page, which is my homepage, but close to 50% of them are dropping off.

Now I can think of a hypothesis of why this is. Maybe my landing page sucks, is unattractive, and there’s terrible content. Or there could be something else, like it just takes too long to load the page and people are not able to get it through to the next level. But we know about 50% of them are moving on to do their first interaction.

A bunch of folks who are coming to my page, then go on to the About Me page, which means something about my landing page was interesting enough to them to make them want to learn more about me. And then they go on straight away to travel from the About Me page. Many also go straight from my landing page to my travel pages, which gives me insights to say that most of my customers are actually relating to my travel content or about who I am.

Photo from Deb’s Blog

Let’s talk about fashion: nobody seems to care. People just drop off. But that’s all the data tells me. So now what kind of hypothesis can I draw out of this? Either my about and my travel content is pretty solid and people want to learn more, and my fashion content is pathetic, so nobody wants to know about my fashion content.

If I really disagree with that assessment, it could be that maybe the page load time for these are different. Maybe the kind of images that I upload for fashion are very heavy, so they’re ruining user experience.

Read next: 5 Reasons Why Product Managers Have to Understand Data

I’ve put this hypothesis out, so let’s test it. Let’s talk about page load time. My starting home page takes 18 freaking seconds. I wouldn’t wait that long. I would move on, but some soldiers hang on and they try to get to my travel. And then my travel load time is six seconds. You can already tell why they have a better experience. Let’s get to fashion—40 seconds. I don’t think anyone actually sat through that.

You can already see how data is helping us test our hypothesis with simple mechanisms that you can use to get to some conclusive evidence.

Cohort Analysis

Let’s dive a little bit deeper into cohort analysis. This is very critical. Not everybody from the United States is feeling the same way about my account, so how do you dissect your user base to study different kinds of users? Cohorts are a way of grouping users into subsets based on some common characteristic. One way of doing that is: let’s analyze all the users that came to my account for the first time, and compare their behavior the next time they come back to my account and interact with it.

Let’s compare user behavior between two different points in time. We do this because as you’re rolling out releases on an ongoing basis, you can test for the exact same characteristic, which is first-time users, and see how different kinds of updates affect first-time user behavior.

Need help prioritizing features? See our template: Product Templates: Competitive Feature Market Analysis

What is the experience like now? Let’s say that I put some really cool content in January and February, and saw that my engagement went up. But then I put some seriously terrible, huge images on my landing page in March, and I see my retention has gone down.

Using first-time user analysis is helpful here because it helps remove those biases of people who have some brand loyalty and will come to back on regardless of terrible UX. I want the first-time experience to compare behavior and reactions to different actions I take.

This is where cohorts come into play. As you’re rolling out features, compare KPIs. First, think about which KPIs you’re measuring. Most commonly, the KPIs that people think about when it comes to cohorts are conversions or retention. That is, are they becoming a paying customer, or are they just coming back to your product itself? In this way, you’ll be able to figure out what features are driving, what KPIs.

Application: Deb’s Blog

Based on the Google Analytics of my blog, I can see in April that of all the users that came to my account for the first time, 3% of them came back the next month. And then half of them came back the month after, and even less the month after that. So in May, it seems like I was able to slightly increase my engagement or my retention. And then it dropped off again significantly in June.

In April, it was a mix of content. But in May I went to Bali and all of my posts on my landing page were about Bali, so I can see why engagement up. In June, only fashion. You can see how I’m trying to correlate between what the content was, and what features really drove people away.


I can take this even further by going into segmentation. Now that you’ve got some support for your hypothesis, how do you get to the next level?

Application: Deb’s Blog

If you’ve heard about the Five Whys, they tell you, don’t stop at, “Oh, I know what it is. It’s fashion.” No, go deeper. Why are people not responding to fashion? Is it just page load time? Why is a page load time so slow? Break it down further. When you think about segmentation, it’s really about slicing and dicing this cohort based on further characteristics, such as device used. We all know that for mobile phones, your bandwidth is an issue, right? It could be that your mobile experience is a lot worse than your desktop experience. This is a hypothesis that we can go ahead and get evidence for, but it could also be other things.

It could be based on demographics. Maybe I had an ad campaign for my blog in April on Snapchat. Snapchat has a lot of young users that interacted with my content. And then in June I had a campaign that was advertising primarily on LinkedIn. Nobody’s looking for my fashion blog on LinkedIn.

Check out: How Product Managers Build A Data Story

This is the way I think about testing how I’m running my campaigns. On my blog, my hypothesis was, I think the experience on mobile is pretty terrible. And I proved that by adding the segment of the device used. In April, out of the six users that came back to my account, nobody came back through mobile, but about 6 of them came back through tablet and desktop. In May. I had 13 users come back of it. Just one of them came through mobile, 12 through tablet and desktop, and likewise in June.

This supports my hypothesis that the mobile experience is worse. Other evidence: average page load time is 17 seconds. But for tablet and desktop, it’s 13 seconds, and for mobile it’s 21 seconds. So this is how you come up with and test a hypothesis using data. It can be something as simple as Google Analytics; run it for about a week and you already have data to play with.

young woman stands in line and looks at phone

On Google Analytics we have segmenting by age, gender, location. If I really want to get into weeds of it, which I should considering I’m having some pretty weak numbers, you can see where my new sessions are coming from, and what my bounce rate. France is seeing a really high bounce straight, so maybe I need to double click and think about why that’s the case. These segments are examples I’m giving you from Google Analytics, but feel free to go to your company and see what tools you guys have for experimentation.

Event-Based Tracking

Next, let’s talk about event-based tracking. This is the most important one that I would emphasize you do, because this is probably what companies do the least. This is where you get into the head of the customer by analyzing every single interaction that a user is having with your browser or your screen without clicks. And the only way you’re going to get this information is by implementing instrumentation.

Your engineers need to think about instrumentation from the time they’re designing their product. It’s a painful process because you’re not just developing code and getting some cool feature rolled out, you’re adding a whole bunch of instrumentation. But keep in mind: if you have to do this post-launch, you’re not being a good Product Manager. It’s a very expensive process to do post-launch and you already lost a lot of really valuable data.

Learn more: Requirements Engineering in Agile: From Myth to Reality

What event-based tracking helps you do is it helps you break down specific parts of the user experience. For example, how many times was an image loaded? How many times did an error message take place, or how many characters did someone type?

An example of this is the address lookup feature when you’re filling out a form; you start typing a few characters and the whole address gets filled up. Let’s say you’re the Product Manager of that feature—what are the KPIs that you want to optimize for?

 It could be you want to minimize the number of characters a person has to type until the right address shows up. And over time you want to see this KPI stay strong, that the number of characters keeps going down as the algorithm gets better. Or maybe, how many times did you see the error Message Address Not Found. As long as those errors are going down, you know that your features are successful.

This kind of analysis helps you ensure that you have a really good user experience. As you build your instrumentation, think about what the customer is doing and how you want to instrument for it. And it’s not always easy to do, some of these are pretty hard. I work with some of our personalization teams, and a lot of these things have technological challenges you have to think about from the get-go. Before you develop the project, you have to think about, What technologies do you use? What does your stack comprise of?

Clickthrough Rate

Lastly, I’m going to talk about clickthrough. Clickthrough is an age-old technique. Marketing campaigns use this a lot, but this is also something that’s useful for all of us to get to the next level of optimizing an experience.

I’ll give you an example. Let’s say you’re signing up for an account and you need to verify through a phone number. You need to enter your number, they’re going to send you a verification code, you need to enter the verification code, and your phone number is confirmed.

If you’re the Product Manager for that, you’re going to see how many users who signed up for this experience actually clicked on “Get Code.” How many of them clicked “Not Now?” If they click “Not Now,” it means your feature is not important enough. Users don’t think that there’s enough value in confirming their phone number, which means you really need to think about what your strategy was. Who are you targeting?

over the shoulder shot of a woman about to click a link on her phone

Let’s say you have a good number of people who did click on “Get Code.” How many of them had to click on “Resend Code,” which means they either never got the code, or they’ve moved on by the time they receive it. You’ve lost the attention of the user, and that’s a pretty terrible experience because there’s a significant churn there.

And now you can tie this together with that initial funnel analysis that we were thinking about. You see churn, but if you measure steps along the way, you can figure out why people actually moved on. These are some really simple techniques to use. You can use all of them, or a combination, to make better product decisions as you go along.

AB Testing

I’m going to briefly touch on AB testing, because AB testing is so essential for Product Managers. Make sure that for every single feature that you release, there is some amount of AB testing that is done, whether you’re a platform team or a consumer-facing team. Check in with your company, they might have some really great experimentation platforms.

Before you release a feature, you want to make sure you can measure interactions. Whether you have eight links on a dropdown menu or three links on a dropdown menu, you need to be able to find out how many people are scrolling to the last link.

Do a test on it, run an experiment. This may seem simple, but these small measurements actually go a long way. Things like, should you have a marketing campaign on your home screen, or a send message to a friend prompt? Essentially, do not release any feature without doing satisfactory AB testing on it.

Using Data in PM Interviews

So how can you get started? It’s very simple. For any of the products that you’re working on, especially if you’re an engineer and you’re looking to move into Product Management, the way you want to fit your story to an interviewer is to start by talking about who your customer is. Don’t talk about what you’re doing right now, because you’re already getting to solutioning first. Talk about who your customer is.

What are their problems? What is the value of the thing that you’re building? Then you talk about how you’re solving that problem. And don’t forget to mention KPIs because that’ll help you discuss how effective your solution is.

Interview Structure

The reason I’m including PM interview tips is because PM interviewing is such a harrowing experience. It’s not fun. Once you have the job, it’s a lot of fun, but going through the process is a little hard. I’m not saying they’re hard problems, but the typical PM interview prompt, “design a product,” there are so many ways to design a product. How do you make sure that you’re able to convince the interviewer you have a great solution?

One really important thing to keep in mind is that a good interviewer is really not looking for the “right” solution. There is no right solution unless you actually go out and test it. What they’re really looking for is structure, your thought process. Are you thinking like a Product Manager?

young person in green sweater talking in an interview

A lot of times interviewer will ask you to pick a product that you want to design. Let’s take an example, something that doesn’t exist. A flying car. Here’s a simple structure that I recommend when you go to an interview:

  • User: Who’s the primary customer/end user
  • Problem: What is their biggest problem/goal
  • Competition & Market Opportunity: What is the market offering today
  • Gap: How can we make a difference
  • KPIs: How can we measure our success

I would recommend walking through this structure. It helps paint a good story. It doesn’t even address solutioning, but it is what will help you get to an actual solution.


So let’s start with the users of our flying car. Who are they? Maybe they’re commuters. They’re wealthy, because they can afford this new technology. And let’s say they commute from San Francisco to Oakland.

You need to define this, because everybody can be a user of a flying car. But when you are designing a product, you’re designing it for a certain segment. You’re talking about commuters because this tells you about their job and what their desired outcome is. You’re talking about their income, which will help you guide your pricing. You’re talking about location because you think that San Francisco or San Jose is probably the hardest commute.

So now you’ve scoped your problem in terms of which specific user segment are you going to solve this problem for.

Read next: Customer Focused Product Management: A Winning Business Strategy


What problems do people in this group have? Clearly money is not a problem. So what problems do they have? Time. So you’re trying to optimize for their commute time.

How do you validate that data? There are some analytical guesses that you have to make. Maybe it takes an hour for people to make it to work and back. So you want to optimize for half of that. These are assumptions that you make and you walk through as you actually develop a product; you prove it based on data.

Competition & Market Opportunity

Now let’s think about the competition, and what your market opportunity is. By market opportunity I mean, is this a big enough market for you to actually play in? Because sometimes you really playing in a niche and it’s hard to justify the success criteria. And it’s fine to play in a niche, but it all depends on what goals you have.

Who’s your competition? Other modes of transport that already exist, basically. What about Uber and Lyft? If the problem you’re solving for is users don’t want to drive, or they don’t want to have to drive on the road, they could be your competition. There is no right answer over here, but when you’re coming up with the competition, you want to think about where are they not playing a role that you can come in and fulfill?


Because the next question that I’m going to answer is, Where’s the gap? If you already have cars, and trains, and planes (which are flying), where can your product play a role? The gap could be that there’s no plane that takes your users from San Jose to San Francisco. They want something that people like them can afford: people who commute to offices, not people who sit in their homes and don’t have to work ever.


Once you identify the gap that exists in the market, that’s when you come up with KPIs. How am I going to measure success? How many people are buying my product? How useful is this product? Has it really reduced commute time, and by how much? Nobody cares about what the car is until you answer all these questions. And once you do, that’s when you can begin solutioning.

Measuring Product Success

Let’s walk through another exercise. How do you measure product success? Let’s pick a different product this time, an existing product: Instagram. What is the goal of the product? Today Instagram caters to many different users, so there are different value propositions for different audiences.

Picking Your Goal

However, when you pick a goal, you want it to be clear so you can determine success criteria based on that goal. One of the most common success criteria, especially for something like Instagram, is user acquisition. So if they already have 2 billion users, the goal could be to grow this to 3 billion users. Or it could be revenue growth; maybe they have a huge customer base, but are they actually converting revenue?

Or monthly active users. This is something that Instagram and all of these social media channels actually do track. Aside from just acquiring users, how much time are these users spending? And you need to think about KPIs for this, because you can have people who are spending five minutes on Instagram in a month, versus those that are spending 25 minutes a day.

Read next: Cracking the AI/ML Product Manager Interview

The latter qualify as more engaged users, so you want to try and bring the five minutes a month folks into higher engagement. You come up with a goal to target and then you can bring them across. That’s why answering this question is really important.

You can also break this down by demographics. It could be that I have great engagement from users in the States, but not in India or Brazil. So then your goal can be, I’m targeting user acquisition from these locations, or of this age group. Why did Instagram introduce stories? To target the age group that uses Snapchat.

person on hammock on instagram

Another thing to think about is conversion. People are paying for your product. A lot of times we should do free proof of concept because people are not willing to trust the product you’re putting out there. Like with Hitachi, because I was supposed to solve crime. They’re not going to just pay money and say, “Yeah, you got it.” They wanted to test it out. We used to do a lot of free POC, and now we want to see how many of those are converting into subscriptions. Or, they start with the subscription, how many of them are renewing?

Referring others is another really good one. We do this a lot for our platform products, actually, because we want more developers to come and consume our platform. We have other metrics that are specific to platforms like total time to market, total cost of ownership, stuff that actually helps you validate the value proposition of your platform. But one really good thing to measure is adoption. How many people are actually adopting your platform, or recommending others to adopt?

And lastly, of course, return on investment. How quickly are you going to see 8X of your investment? I remember whenever we had these P&L sheets, the Hitachi execs used to always look at the last square on the Excel sheet. Tell me when I’m going to hit twice of that, that’s always their question.

So ROI takes into consideration acquisition costs, support costs, delivery costs, professional services costs. And how are you going to overcome that to become that hockey stick growth goal that we all talk about?

Improving Your Product

How do you improve a product to get to that? Let’s take Instagram. Can you think of ways to improve the product? It’s pretty good. Let’s start with the goal: connecting people. What are the criteria for this metric? It could be that it’s connecting a lot of people in the United States, but not connecting enough people in India. That’s an area to improve your product.

Now think about what are the smaller metrics you can improve to achieve that goal. The user base is not growing, indicating that people in India are not using it. Or revenue is not growing; people are using it, but nobody’s actually converting. This means the ads are not working, so you need to do better targeting of those ads. Do we need higher engagement? That is, people have signed up, but nobody’s logging back in. That means the content that you’re putting out there is not relevant enough. You can talk about click-through rates.

You might also be interested in: Marginal Improvement Through Analytics with Spotify Data Scientist

The next thing you think about is competition, always bring competition into the picture. Maybe there’s another Instagram, there’s a local app. It’s so true of China, for example, they have their own social media channels. And that’s why some of our social media platforms are not successful there, because there’s an existing competition that has higher market share. What are they doing better? Where’s the gap? That gets you to your solutioning and the areas that you can actually focus on improving.

And don’t forget, many of you are talking about an existing product. Think about how you’re going to roll this out. If it’s a cloud-based product is probably easier, but if it’s actually something that needs to be installed, think about what the delivery plan needs to be, what your professional services costs need to be.


Lastly, how do you test success? Talk about the metrics. Talk about doing usability or AB testing. Talk about feedback. Talk about learning, not just through data, but through action. Once you’ve learned more about your hypothesis, go back and validate. Talk to those customers again, do mock-ups, whatever it takes to understand if this is really working for them.

Think about support metrics as well, because many of these organizations have a significant investment to grow their global ops teams and they’re not sure how to tie their support investments back with their product growth and development costs.

Final Tips

When answering interview questions, throw in some tools that you have played around with. It could be something as simple as what I did with my blog. This is great because if you actually have some hands-on experience with using analytical tools, it will take you a long way and give you some authenticity when you’re talking about this.

Logging is also really important. Logging is close to what I meant with instrumentation; it’s where you can log the values of different counters. Things like how many error messages showed up, how many times would this AJAX call take place, etc. This will help you a lot in testing how successful your launch was.

Use these structures to practice through some of your interview questions until it becomes a part of your mindset. All of these questions are related—they might sound different, but they all are actually talking about the exact same thing, the same structure, and you’ll hear various permutations and combinations of these questions. When you go to your interviews, they’ll never be looking for a different kind of answer. Every interview is looking for this exact step-by-step thinking of keeping it consistent, keeping it succinct, and getting to goals, users, problems, solutions.

Enjoyed the article? You may like this too: