Product Launches and Risk: A Tough Teacher

I think about risk because I’ve seen firsthand what happens when you don’t: rattled product teams, irate stakeholders, and frustrated customers. My very first product launch had all of this and then some because our team committed to delivering a complex product on a hard deadline with no internal experience launching a product in that category. We didn’t appreciate that ours was a risky venture and paid for it.

But I also think about risk because I’ve seen what happens when you plan for risk in advance of launches: phased releases, fallback plans, and other forms of risk mitigation that save the day when things don’t go as planned. In a high stakes web replatform at Barnes & Noble, our product team decided to slowly phase in traffic to our new site. When our new site showed issues on launch, the phased rollout minimized the number of customers impacted and gave us the cover we needed to make changes that led to product success.

In this blog post, I’ll cover questions newer product managers can ask to assess the risk level of their projects and as well as risk mitigation strategies that can be implemented as needed. If there’s one thing I want you to take away from the article, it’s that product managers who manage risk ultimately manage their way to better outcomes.

What is Product Risk and Why Do PMs Need to Own it?

But before we dive into assessing risk, what is in product management and why should PMs care in the first place? In product management, we loosely define two categories of risk: risk, related to our ability to successfully deliver a product from a technical and operational perspective, and market risk, related to your product’s ability to generate the successful outcomes after delivery.

When a team pushes a feature to production, without any critical bugs, that as expected, they’ve managed risk and built their product well. When that feature then proves successful from a customer perspective (customers use it and love it, new revenue exceeds expectations, etc.), the team has managed market risk and built the right thing.

Managing these two types of risk is a team task, but product managers must lead the charge. Why? Because at the end of the day, product managers are responsible for the outcomes of their launches. When the team ships a feature that causes customer issues, we don’t point our fingers at the developer who wrote the code or the QA engineer who failed to flag the bug. We take responsibility for the damage and learn from it. As a PM, if we care about outcomes, both for our customers and our business, we must also care about risk.

Questions to Assess the Risk Level of Your Project

Up until now, we’ve spoken about risk theory, a nebulous trickster that threatens to derail our launches. But how do we make it real, analyze it within the context of our own projects, and then deal with it? Here’s a series of questions to ask about your project, along with suggestions to mitigate the risks you uncover.

1. How confident are we that we’re building the right thing?

What’s your confidence level that this product will [insert desired KPI/outcome]? What makes you that confident? What could we do to increase our confidence?

When answered truthfully, the responses to these three questions help product managers assess and mitigate market risk by revealing a team’s potential overconfidence in their product.  When we build something that is a basic but clear request coming through customer service, our confidence level should be high that our product will hit the mark with customers. But when we venture into new territory, initiating a project championed by a powerful stakeholder and backed only by anecdotes, our confidence should be low and we should actively try to mitigate the market risk we just identified. When your team shows low confidence, consider:

• Additional user research to vet the idea with target customers

• Combing through current product analytics to see if data on an existing feature might help the team intuit how your new product might perform.

A colleague once told me that the main job of a product manager is to de-risk the company roadmap.  When you increase your confidence or invalidate bad assumptions before building and shipping a product, you’re doing just that

2. Has the team launched something like this before?

Is this a routine release? Does the team know with precision how long development will take because they’ve done it before? Is the infrastructure on which the team will develop well-worn and documented internally? If your answers are yes, congrats, your executional risk is low.

But if this is a pioneer project for your company, outside of the team’s realm of experience or expertise, balance the excitement of launching something new with a cold dose of reality. This project is risky and needs to be managed as such. Here are a few mitigation strategies:

• Test the project’s feasibility with proof-of-concepts and spikes.

  If there’s any doubt that the team will actually be to accomplish the project technically, rather than diving right into the work, give the team the time it needs to develop proof-of-concepts and complete research to increase their confidence in execution.

• Resist hard dates and bake extra time into the plan.

  When launching a new product, no (project) plan survives the battlefield. Factor the unknown-unknowns that come with brand-new initiatives into your plan and ensure the team has more than enough time for development and testing. Narrow in on a date only when the project is well underway and confidence has increased.

3. How “big” of a release is this? 

Are you updating core parts of your product, or a lonely feature buried in a menu that isn’t mission critical? Do all customers interact with this functionality, or only a small subset of users? By asking these questions, you’re sizing the downside if things go wrong. If you’re launching a product to a majority of your user base or making changes to core functionality, consider:ff

• Internal testing periods, in which team members and stakeholders test the final product in production before customers get access.

• Phased rollouts (BETAs, split traffic launches), where you release your product to a subset of users before it moves to general availability.

• Off-peak releases to limit the impact to as few customers as possible if issues arise.

These suggestions may seem basic, but I continue to see teams launch product updates to core features an hour before marketing drives traffic to their products. There’s a reason midnight launches are a thing in technology and that you should stay up late to release when making “big” changes to your product.

4. Can we roll-back if we have to?

In his 1995 letter to stakeholders (see “Invention Machine” section), Jeff Bezos writes about Type 1 decisions (irreversible, unable to return to the previous status quo) and Type 2 decisions (reversible, impermanent). I like applying his thinking to product launches. In a Type 1 release, a rollback of the launch is not possible, either for technical or business reasons. In a Type 2 release, if the launch is disastrous, the day can be saved by technically rolling back the feature, at least temporarily. 

• Ask your team whether the project can be implemented behind a feature flag, allowing the team to quickly turn off the feature in production without writing additional code.

• If your launch truly is irreversible, bake extra testing time into the schedule and deploy off-hours, buying the team to make necessary changes if the initial launch reveals issues.

A Little Paranoia Is a Good Thing

It’s no fun to talk to the team about the potential of rolling back months or even years of hard work. But neither is a firestorm in production you didn’t prepare for and can’t resolve quickly. As product managers, we push forward by launching new features while appreciating the risk of our endeavors. With just a little paranoia and attention to risk, we generate the best potential outcomes for our customers and companies. 

Meet the Author  

John Kresse is a Senior Product Manager at SoulCycle, where he launches products for SoulCycle’s mobile team. Previously, he led mobile e-commerce initiatives at Barnes & Noble and got his start in product developing education technology at W. W. Norton.

John loves customer research and uncovering the “jobs” his customers need to complete as well as digging into analytics to quantify customer behavior and identify areas for product improvement. You can find John with his two wonderful children, on a bike at SoulCycle, or on the basketball court.