How to Test Your Product Ideas by SoulCycle VP of Product

In order to increase the certainty of a product, it is important to leverage different kinds of tools, each one is unique and tailored to different kinds of situations to achieve a wide range of goals. Which one is the right for you?

Meet Tim Holley

Related image

Tim Holley leads the Product Team at Soul Cycle which is a New York City-based fitness company with 88 studios in the US and Canada. He has a Masters in Management Science and Engineering from Stanford University. Before working at SoulCycle, Tim worked at Etsy where he made significant contributions to the company. He helped Etsy grow from $525M in GMV in 2011 to $3.2B in 2017 by delivering impactful products.



Existential Questions

  • Where are you going?
    • Think about product strategy, vision, and principles.
  • What does great look like?
    • Set a goal and work towards it.
  • Do you have a line of sight to achieving your goals?
    • This plays a very important role; don’t put all the eggs in one basket; use a portfolio approach.
    • For example, when you’re starting a project, the budget of the project and the certainty of success should be inversely proportional; for a $1M project, the certainty should be around 20%.
  • How do you increase certainty?
    • Pragmatic > Dogmatic: You have to be Pragmatic versus Dogmatic. In Etsy land, nothing went out without being A/B tested, they were so incredibly dogmatic that if you couldn’t A/B test it, it wasn’t really a thing. And that is probably and intellectually the right thing because you can really quantify the impact. But in reality, it slows you down and it’s potentially limiting because launching a new feature and A/B testing a new feature is really difficult. 
    • A/B testing is not cheap and sometimes it’s not the answer. A/B testing is just not the silver bullet that delivers answers to you. But if not that then what? The answer is below. 
  • Use a set of tools to increase certainty in different situations
    • For example, you can use the “Pilots/Betas” tool on a sample of 100+ people and can get feedback on product usability and understanding of the infrastructure.
    • You can use the “1:1 Interviews” tool on a sample of 20 people and get potential directions on where to head.
    • Other tools are “User Tests”, “Surveys”, and “A/B Tests”.



  1. Understand what gets new riders hooked
    1. For example, if the context is “reimagining an existing flow” and the goal is to get “more new riders hitting 5 rides”, then the tools used are 1:1 interviews, user tests, and A/B tests.
    2. Dive deep into people’s habits, create archetypes, develop provocations, narrow down the direction.
  2. Enable Sellers to Offer Digital Items
    1. For example, if the context is “Building new B2B tools” and the goal is to “increase GMS from digital items”, then the tools used are 1:1 interviews, pilots/betas, and A/B tests.
    2. Understand current behavior/issues, test basic tools with a small pilot, grow pilot to increase inventory, and provide A/B buyer-side experience.
  3. Validate guest checkout, without the guest part
    1. For example, if the context is “optimizing an existing flow” and the goal is to “increase conversion rate”, then the tools used are user testing and A/B testing.
    2. Inspect the funnel, find the pain points, do a lot of thinking, grab some people for hallway testing, and launch the simplest thing possible.

Do you test products ideas differently? Have you tried one of Tim’s suggested ideas?

Enjoyed the article? You may like this too: