In order to increase the certainty of a product, it is important to leverage different kinds of tools, each one is unique and tailored to different kinds of situations to achieve a wide range of goals. Which one is the right for you?
Meet Tim Holley
Tim Holley leads the Product Team at Soul Cycle which is a New York City-based fitness company with 88 studios in the US and Canada. He has a Masters in Management Science and Engineering from Stanford University. Before working at SoulCycle, Tim worked at Etsy where he made significant contributions to the company. He helped Etsy grow from $525M in GMV in 2011 to $3.2B in 2017 by delivering impactful products.
- Where are you going?
- Think about product strategy, vision, and principles.
- What does great look like?
- Set a goal and work towards it.
- Do you have a line of sight to achieving your goals?
- This plays a very important role; don’t put all the eggs in one basket; use a portfolio approach.
- For example, when you’re starting a project, the budget of the project and the certainty of success should be inversely proportional; for a $1M project, the certainty should be around 20%.
- How do you increase certainty?
- Pragmatic > Dogmatic: You have to be Pragmatic versus Dogmatic. In Etsy land, nothing went out without being A/B tested, they were so incredibly dogmatic that if you couldn’t A/B test it, it wasn’t really a thing. And that is probably and intellectually the right thing because you can really quantify the impact. But in reality, it slows you down and it’s potentially limiting because launching a new feature and A/B testing a new feature is really difficult.
- A/B testing is not cheap and sometimes it’s not the answer. A/B testing is just not the silver bullet that delivers answers to you. But if not that then what? The answer is below.
- Use a set of tools to increase certainty in different situations
- For example, you can use the “Pilots/Betas” tool on a sample of 100+ people and can get feedback on product usability and understanding of the infrastructure.
- You can use the “1:1 Interviews” tool on a sample of 20 people and get potential directions on where to head.
- Other tools are “User Tests”, “Surveys”, and “A/B Tests”.
- Understand what gets new riders hooked
- For example, if the context is “reimagining an existing flow” and the goal is to get “more new riders hitting 5 rides”, then the tools used are 1:1 interviews, user tests, and A/B tests.
- Dive deep into people’s habits, create archetypes, develop provocations, narrow down the direction.
- Enable Sellers to Offer Digital Items
- For example, if the context is “Building new B2B tools” and the goal is to “increase GMS from digital items”, then the tools used are 1:1 interviews, pilots/betas, and A/B tests.
- Understand current behavior/issues, test basic tools with a small pilot, grow pilot to increase inventory, and provide A/B buyer-side experience.
- Validate guest checkout, without the guest part
- For example, if the context is “optimizing an existing flow” and the goal is to “increase conversion rate”, then the tools used are user testing and A/B testing.
- Inspect the funnel, find the pain points, do a lot of thinking, grab some people for hallway testing, and launch the simplest thing possible.
Do you test products ideas differently? Have you tried one of Tim’s suggested ideas?