Prioritizing Roadmaps Based on Data by Amazon Senior Product Manager

Building the best product roadmap is hard, harder still to get everyone aligned on it. That’s why you need data. Amazon’s Senior PM, Artem Fedyurin, joined us to talk about how to build roadmaps based on data, and walked us through the most useful prioritization techniques.

Meet Artem Fedyurin

Artem Fedyurin

Artem Fedyurin is a Product Manager of Marketing Automation at Amazon. He is currently responsible for building tools that enable Amazon marketers to reach millions of customers through on-site and email channels at scale. Prior to his current role, he was a Senior Business Manager of Device Market Intelligence and Strategy at Microsoft.

Artem has extensive experience in Product and program management, strategy, investment banking, and finance. Moreover, he graduated from the University of Michigan with an MBA. Artem is a fast learner, creative problem solver, and hard worker that always finds a way to get the job done. 

Prioritizing Roadmaps Based on Data

Before we jump into roadmap prioritization, let’s talk about what roadmap is and its goals.

First of all, a roadmap is a communication tool that helps a lot of stakeholders around the common goal. So if you have multiple stakeholders, customers, clients, rather than helps them understand what you’re working on and what your end goal is.

Second, it helps to set a direction of the product development in accordance with the org’s goals, and should be closely tied to the vision and the goals of the organization. And as a result, it shouldn’t be rigid. It should be flexible. It should rally around your priorities and be able to change based on the change in core priorities. Roadmaps should never be set in stone and should change as needed.

And last but not least, it should get customers excited, which means if you have customer support, this eventually will translate into a faster product adoption. So the roadmap should get customers excited.

Roadmap Focus vs Product Type

Next, let’s talk about the roadmap dependency on the product type.

So there are different types of products and depending on what type of product you have, your product roadmap will vary. So growth versus volume. As you know, all products go through a product life cycle. In the beginning, when the product is in a growth stage, the roadmap should focus on growing a user base, growing adoption, et cetera. So think about a social app in the beginning. Any social app focuses on features to increase the customer base, and as they become bigger, they start focusing on monetization and increasing value to shareholders.

You might also be interested in: Product Management Skills: Stakeholder Management

The second type of product is B2B versus B2C. So B2B is business to business. An example of a B2B product could be a CRM app by Salesforce. For example, in those cases, the product manager prioritizes the roadmap based on the use cases of a smaller set of larger clients who are companies/other businesses.

B2C is Business to Consumer. So an example of a B2C is Netflix or Google Maps that you use as a consumer everyday. And for B2C products, there is a much larger customer base. So think about Facebook, it has billions of customers and they should focus on customer groups or customer cohorts with similar goals and similar needs.

The third type of products could be internal versus external. So internal products are products that are built inside their organizations. So as companies grow, they may need internal products. For example, many large tech companies have their own product for HR and for accounting, for operations. So those products would have a smaller user base and might focus on things like process automation and optimization or efficiency, improvement, and external products. These are products built for external customers and organizations and are focused on driving and increasing usage and engagement.

Often all of these three product types can overlap. And as the companies grow, the companies change, they would move from one category to another. For example, in the beginning, Facebook was purely a growth, B2C, external product. However, as the company grew there became a volume product, focusing not only on B2C, but also on the B2B segment. And additionally Facebook is used internally for internal communication, document sharing, et cetera. So it has both internal and external elements.

So as you can see, Facebook expands across all of these three categories.

Where Do Feature Ideas Come From?

The next question is where the feature ideas come from. And there are plenty of sources and depending on the product type, as described earlier, the sources might include the following.

  • PM’s vision – how the PM thinks about the product.
  • Customer requests – done through in-built feedback mechanisms or through customer feedback. If it’s B2B then those requests are given directly to the development team.
  • Partner requests –  often used by B2B teams.
  • Sales requests –  an example where you have a team of salespeople, they call different partners trying to sell your product, and some of those partners, so customers can provide feedback.
  • Customer support requests – when a customer called your support team complaining about something or wishing to have additional functionality.
  • Competitor features – if your competitor is building something, you might also consider adding it to your roadmap
  • Leadership requests – ideally they should align with the product vision, but also can include more strategic requests in organizations that have more than one product.

What Is Roadmap Prioritization and Why Does It Matter?

Let’s talk about what is product roadmap prioritization and why it matters, and we’ll talk about different types of products and roadmaps.

Prioritization is done by a product manager, and is needed to define which features to build and which ones not to build. And interestingly enough, the most difficult part is oftentimes to say no to features because many features might have value to the customer, but the value can be different. And that’s where prioritization comes into the game.

So a product manager needs to decide which features to build, which features to keep in the backlog, which features not to build. And within those features that PM decides to build the product manager needs to decide which features to build first and which features can be built last.

Prioritization matters first of all because of limited resources. Every organization has limited resources, even companies like Amazon, Microsoft, Google, you name it. They all have limited resources because of the very large number of ideas that all these companies have. And this means that development hours are limited. So limited resources, in the case of tech products,  means limited development hours. It becomes very important to optimize the return on the resources that the work invests in.

Secondly, different features have a different value to customers. So some features are crucial for the product adoption while other features are just nice to have.

Thirdly, given the limited resources and different value to customers, products that have better prioritized features, we’ll get better traction with customers and can get a competitive advantage if they released more important features earlier to the market.

MoSCoW

Prioritization frameworks are used by product managers and tech companies.

There are simple and more sophisticated prioritization frameworks that exist in the industry. 

The first framework I’m going to talk about, which is a relatively simple framework called MoSCoW. MoSCoW is an abbreviation of feature categories that are used to describe the relative volume of a feature to the user. The categories include:

  • Must Have
  • Should Have
  • Could Have
  • Won’t Have

The PM collects feedback from users, sales, et cetera, on about the relative value of all the features on the roadmap and decides which features to put in which category buckets.

And usually an MVP will be released with all the Must Have features, and Should Have features would be released as a fast follow to the Must Have features. Could Have features might be released later in the product life cycle. And Won’t Have features will not be developed.

MoSCoW

So for example, here’s an example of a prioritization for an internet video service. So imagine you’re working on a product, something like Netflix, and you might have the following features in your backlog. So the first would be to watch movies. So customers should be able to watch movies on your video player. And then customers should be able to search movies in your app. Customers want to be able to rank movies and customers want to have multiple profiles. So how do you prioritize these features using the MoSCoW framework? Well, you would put them in the different buckets.

Watch movies is probably a Must Have, so you cannot release a video service without the functionality enabling users to watch movies. The same is true about search movies. If you have Netflix and you cannot search movies, it’s not a great experience. You will have to scroll through thousands of movies to find something to watch.

Check out: 3 Prioritization Techniques All Product Managers Should Know

Rank movies on the other side, is more optional for MVP or minimum viable product at least. You can put it in the Should Have maybe Could Have category. And then the ability to create multiple profiles, not very crucial for the first version of the product, so you could put it in the Could Have category. This is a very simple, pretty subjective framework, and usually used for fast prioritization.

Desirability, Feasibility, and Viability

The next one is a little bit more sophisticated privatization framework called Desirability, Feasibility, and Viability.

So the desirability criteria is meant to answer, do customers want this feature and do competitor products have, or are competitors building this feature? And so this is a customer focused criteria that you want to take into account when prioritizing features.

The next one is feasibility. So the feasibility criteria helps answer questions like, can we build this feature? Do we have technical expertise to build this feature? It’s a company-focused criteria.

The third one is viability, and that focuses on the business side of the feature, and it’s meant to answer questions like, can we make money off of this feature? Or will it generate the desired outcome for us? And this is a business-focused criteria. Using this framework, product manager’s prioritize each feature on the roadmap, score them for desirability, feasibility and viability, and then use some of the scores to rank the features.

Prioritization Frameworks: Desirability, Feasibility, Viability

So you can see there are three overlapping circles and of the venn diagram. And ideally your product should be somewhere in the middle, in the intersection of all of these three circles. That means that it meets all the criteria for desirability, viability, and feasibility.

For example, as a product manager for a music service, for example, Spotify or Pandora, you could use these frameworks to prioritize features on your roadmaps. For example, let’s say you have three features that you want to prioritize. First one is search songs. Second one is shuffle songs. And third one is to see all artists’ albums. For search songs, you can say that it’s very desirable by customers. Again, having Spotify without search would be not an ideal experience in terms of visibility. You can say it’s also three on a scale of zero to three.

You can say it’s also three because our team is building Spotify, so presumably we should have engineers who can build the search functionality because it’s not something unusual these days. And so viability is also three because if our competitors have this feature and we don’t, we could probably lose customers to competitors.

So by building this feature, we’ll have more customers, so it’s good for business. And so by adding all these three scores for this ability, feasibility and viability, you get the priority score. So in this case, using this approach, the score is nine. You go through the same exercise for shuffle song, see all artists albums, and you’d get scores of eight and seven. And that’s how you rank the features. So search songs will be writing the highest, then goes shuffle songs. And the third is see all artists albums feature.

ROI

Now we can move to the next one. So ROI means return on investment. You’ve probably heard about this one. Product managers need to assign the return on investment value to each feature on their own. So return is a revenue or other kind of benefit. It could be a revenue for external products, but also could be time savings. For example, cost reduction for internal products. If the product is focused on process automation and investment in this case is development resources. Which includes the cost of a developer working on the feature, and also any additional services that might be required for this feature. For example, additional cloud computing services.

So a product manager assigns dollar values to each feature, and dollar values for return, and dollar values for investment. For example, you could use this same prioritization framework to rank the features for internet music services such as Spotify.

Check out: Product Management at Spotify with Former VP of Product

For example, you want to prioritize features on the roadmap and you have a feature – search songs. So how do you estimate the revenue and dollar value for the revenue? You can do either financial modeling focused on, for example, increasing the number of paying customers, thanks to the new feature. If you can estimate how many new customers you can get from building this new feature and how much revenue you can generate per customer per year, then you can calculate projected annual revenue.

In some cases, you might want to use some statistical analysis, such as conjoint analysis, which is based on the customer survey. And it assigns different values to different features based on the customer, willingness to pay…

Whichever method you use, you assign the projected revenue, then you estimate the dev effort and you assign the dollar volume to that too, by just multiplying the number of dev hours needed to build this feature by the cost of an hour and a half in your company.

And then you add any other costs associated with building this feature. Again, this could be cloud computing service, for example, or new servers or any additional equipment needed to build this feature, et cetera, and calculate ROI.

You simply take the projected revenue and divide it by the sum of all your costs. For example, the ROI for search songs will be 200,000 over 60,000 in cost minus one. And that gives you the ROI of 233%. And you do the same calculation for other features on your roadmap, like shuffling songs or artist’s albums, depending on the ROI.

So you can see that using this prioritization framework, it’s ‘search songs’ that will be around the highest, ‘see all artist’s albums’ will be around the second, and ‘shuffle songs’ will be around the third.

ROI Scorecard

There is an alternative to our ROI model based on dollar values, and using this framework you estimate the relative value of the benefits and the relative value over the cost of all the features. So benefits are tied to the company goals. Then relative value is the same, depending on how the particular feature will impact the goal. And the numbers can be both positive and negative. So negative numbers would mean that the feature negatively impacts the goal. For example, consider a grocery delivery startup like Instacart. So for those of you who don’t know, Instacart is a delivery startup that doesn’t own stores, but instead offers grocery delivery from other chains, such as Costco, Safeway, et cetera. And so as a product manager on Instacart, you could use this privatization framework to prioritize these three features.

  • Offer two hour delivery
  • Wishlist
  • Add items after placing an order

So how would you run the prioritization in this case?

ROI Scorecard

First you create columns for your goals, for what your company or for what your organization wants to drive. So let’s say in this case your goals include increasing the number of customers, increasing average checks, and increasing profit. And so your cost will be dev effort.

So let’s consider an example for offering two hour delivery. So let’s say we build this feature into our two hour delivery. Most probably the amount of customers will go up because anyone would love this feature. However, as you build this feature, your customers will be willing to place orders more often because now they don’t need to build a bigger cart. They can place an order and it will be at their door in two hours.

So it requires less planning on the customer side. Most probably the average check will go down. And that’s why you might want to put minus two in this case, which means it will negatively impact your average check goal. The same is true for profit because once you build this two hour delivery feature, there will be a smaller check per delivery, which means your cost per delivery will be higher relative to your revenue per delivery, which means your total profit will go down. And essentially your cost will increase as you’ll need to do more deliveries because customers will start ordering more often. But the revenue may stay the same and not go with a number of deliveries. So that’s why you would put a negative number for profit.

And your dev effort, it should be pretty easy to add this functionality to your app. You calculate this score by just summing up all the benefits and dividing it by the effort. So in the case of offering two hour delivery, it will be 5-2-2, which gives you one over one.

You can go through the same exercise for wishlist and add items after placing an order.

Learn More About Prioritization

Interested in learning more about prioritization? We’ve got even more great talks over on our YouTube channel! Try these ones out:

Product School Pro

Enjoyed the article? You may like this too: