A/B testing is the most common approach used to validate optimization hypothesis. This approach while powerful has many limitations and does not always result in the highest ROI. Personalization is an alternative approach which can maximize value for specific audiences but can be costly to implement at scale.
In this article, Ali Daftarian explains the strengths and weaknesses of each approach that will be critical in forming a winning optimization strategy. Check out the full video below.
A Yogi in FinTech Product Management
Ali Daftarian is currently the Product Manager in mortgage lead gen at Bills.com. Prior to this, he worked as a Product Manager at Western Union & Simplikate (later acquired by Phunware). Ali has a Bachelors in CS from UBC and an MBA from SMU (in Canada). He is also a certified Yoga Instructor and teaches yoga during his free time.
A Case Study: Mortgage Lead Gen Flow Overview
Ali’s responsibility is very similar to other PM’s mission: to enhance results while limiting costs. Let’s see what the current mortgage lead generation flow looks like.
- Focus on a single transaction (‘lead’).
- Sensitive to market conditions (Ex. Increase in interest rates).
- Impacted by government regulations (Ex. HARP extension).
- Marketing-driven (Ex. changes in marketing costs).
- Highly competitive and dynamic financial market.
- Lead buyers exert great influence on the quality and price for leads they accept.
As you can see, there are many dynamics involved in obtaining a single lead; and more could be added. The product team needs to find ways to isolate variables and conclude whether one of the unknown factors can be isolated. There are two product optimization tactics that they can use: A/B Testing and Personalization. Let’s see which one can help in this situation.
Product Optimization Tactics: A/B Testing vs Personalization
A/B testing is a way to compare two versions of something to see which one performs better. It is one of the most important procedures and is recommended by Ali for testing out major fundamental changes in the product. For example, Bills.com used A/B testing when they made changes to their UX.
- Allows for distinction between causation and correlation.
- Streamlined and easy to use tools available (ex. Optimizely).
- Can yield significant improvements to KPI goals.
- Does not maximize value across audiences.
- Not easily adaptable to changes in user behavior or market conditions.
- Can be time consuming to get statistically significant results and business value.
- Use Cases
- Appropriate for testing fundamental changes in the online experience.
- Test difference in number of steps and grouping of questions for online forms (Ex. LMB).
- Test fundamentally different designs or user experiences (Ex. LMB).
- Longer forms and moving deeper in the funnel (Ex. LT).
Personalizationis the method of customizing a service or a product to meet the specific needs of a segment of consumers. For example, when a user lands on your website and searches for a specific term, you need to make sure that the relevant headings appear in your advertorials or forms.
- Maximizes the business value from specific consumers’ groups.
- Presents the best customer experience to specific consumers’ groups.
- Difficult to discover differences in audiences.
- Costly to implement personalized experiences at scale.
Predictive Personalization is the method of using predictive technologies like Machine Learning to analyze trends in data sets and predict what your customer wants based on the trends. It also helps determine the variations of the product or service which performs best for specific consumer groups.
- Maximizes the business value across all consumer groups.
- Presents the best customer experience across all consumer groups.
- Fewer commercial tools are available.
- Costly or complex to implement technology in-house.
- Ali explains an experiment his team conducted to increase the form completion rates of the user on their website. In their experiment, they defaulted certain fields of the form and had a bigger ‘control to action’ button. This led to increased form completion rates on their website.
- Other examples include using engaging headlines on landing pages, having convenient ‘common’ answers, setting expectations and progress indicators, and providing helpful tips.
Choosing Product Optimization Tactics: A Summary
Considering the many complex decisions that PMs face every day, optimization techniques can help a lot. They provide data-driven methodologies to make choices which, even if they prove to be wrong, can help you learn in the long run. Check out this summary:
- Use A/B testing to find the best base site experience.
- Use personalization to optimize for different audiences.
- Leverage predictive personalization if available to:
- Maximize business value and consumer satisfaction.
- Adapt to changes in market and consumer behavior.
- Streamline ongoing optimizations.
Here are more details on optimizing your Product Management department. Check it out!
This post was adapted from content summarized by Varsha Jayaraj