Being an AI Product Manager

Editor's note: the following is a guest blog post from an external contributor

In recent months, the transformative power of Artificial Intelligence (AI) technology has revolutionized the accessibility, understanding, and analysis of information. Businesses now operate differently, innovate more effectively, and solve customer problems with the aid of AI. This paradigm shift presents an important career question for Product Managers: How can you become an exceptional AI Product Manager? 

Drawing from my experience as a Product Manager at Microsoft, DocuSign and as a startup founder, this blog post aims to shed light on the essential topics to master in order to excel in this role. But before we jump in, let’s pause to address a crucial question that I get asked quite often by people aspiring to be an AI Product Manager.

Q) As an AI Product Manager, do I need to understand the Technology behind AI?

Often, Product Manager roles are described using a Venn diagram showing the intersection between UX, technology and business. While implementing the technology itself is not generally a Product Manager's primary focus, for an AI Product Manager, a high-level understanding of the technologies powering AI can greatly benefit you.

By grasping concepts such as AI model functioning, model training, inference, and performance measurement, Product Managers can effectively collaborate with their engineering counterparts. 

My guidance here would be to take a quick course that teaches you the basics of Artificial Intelligence. 

Now, assuming you have this basic understanding of AI, let’s dig right in. 

Exploring the Concept of AI in Couples Therapy: 

To guide our discussion, we’ll explore the application of generative AI in couples therapy. With Large Language Models (LLMs)taking centre stage in today’s media, it’s an intriguing question to ask - can I make a product that can replace my expensive couples therapist? 

1. Start with the Problem - Not the solution 

Just as with any other Product Management role, an AI Product Manager follows the same paradigm: your primary responsibility is to identify the customer’s pain points.

Technologies such as large learning models (LLMs), machine learning, artificial intelligence, and blockchain are merely tools to address these problems. 

However, the flip side of AI's capabilities is that it can now uncover and solve problems that may not have been apparent before. This highlights the importance of thoroughly understanding the magnitude of the problem and assessing whether solving it would lead to growth opportunities. In other words, conducting customer research becomes even more crucial. By engaging with customers, gaining insights into their needs, and evaluating the potential impact of addressing those needs, you can make informed decisions about which problems to prioritise and how to leverage AI effectively. Remember, customer research plays a vital role in aligning AI solutions with market demand and maximising the growth potential of your products.

For the couples therapy example, I did a quick user study spread across 15 couples at various stages of their relationship. I asked each couple about 20 questions. What did I learn? Most couples are spending a considerable amount of time and money on couples counselling. While almost all of them find counselling useful, very few of them find it affordable even with insurance covering part of the costs. So the next question we need to ask ourselves is: Can we make a solution that is readily available, affordable and just as good as a counsellor? Do we think AI, and LLMs more specifically, can be used to solve this problem? I’ve tried to answer this question in the next few steps.

2. Do I have the necessary Data? 

Successful AI Product Management revolves around identifying and utilising the right data to create innovative products that captivate customers and foster long-term engagement. It's important to consider the strategy of sharing Product data with large-scale cloud providers when utilising public large learning models (LLMs), as data can serve as a competitive advantage and sharing it may lead to a disadvantage in certain cases.

Many AI capabilities often lack labelled or annotated data, making it crucial to prioritise items based on data availability. For areas that require more annotated data, it's essential to devise strategies that allow for obtaining additional annotated data or leveraging publicly available data partially.

Furthermore, it's vital to have a clear understanding of how and if customer data will be used for training models. This involves anonymizing the data and obtaining explicit customer consent.

Now, let’s go back to our couples counselling app. Here are some important considerations to think about before saying that you can build an AI solution:

1. Is there any data related to couples counselling that can be used to train a model? In this case, the answer might be yes.

2. Can data be collected from customers to enhance model efficiency? It's worth noting that most customers may be hesitant to release their personal data to an online service.

Looking at the answers above, one might strategize to use AI and develop a private LLM model that does not share data and only uses anonymized customer data for training. 

By addressing these questions and taking appropriate measures, Product Managers can navigate the challenges of data availability, privacy, and consent while leveraging AI.

Now let’s say you’ve worked with your development and user experience team and made a generative AI solution that can partially replace a therapist. The next question you should ask is:

3. How do I measure success? 

This is an important question you'll face and is relevant for you to answer for the success of your product. However, measuring the success of AI features requires a different approach compared to traditional metrics like CSAT, PS scores, and CTR values. While measuring performance of a model using Precision, Recall and F1 scores is important, there are additional metrics especially in the case of LLMs which are also important.

Here are some additional metrics that I think are helpful to know as an AI Product Manager:

1. Hallucination Metrics 

For this, analyse the generated output and compare it against the ground truth or reference data. Identify instances where the generated output contains information that is not present in the ground truth.

So for our example – For a given couples counselling question, define certain entities that are required in the answer for it to hold true. 

Now measure times when none of these entities show up in the final answer given by the LLM. This will tell you when the model is mostly making up answers.

2. Semantic Similarity against human generated answers

Measuring semantic similarity in AI models involves assessing the degree of similarity or relatedness between two pieces of text. In this case this would measure text between output generated by LLMs as compared to what a human would have responded with. 

This helps us understand how close the answer was to a human response.

3. Latency (p75,p90,p99)

Latency is an important metric for assessing the efficiency and responsiveness of AI models, especially in real-time or time-sensitive applications. As a Product Manager, you will need to make sure your models respond fast enough or else most customers will lose interest.

Remember every feature developed, including AI features, should align with larger end goals. Whether it's driving usage, revenue, downloads, or other metrics, Product Managers must always connect their efforts to tangible outcomes.

Conclusion

In conclusion, as an aspiring AI Product Manager, continuous learning, adaptability, and staying attuned to customer needs and technological advancements are key to navigating the dynamic AI landscape successfully. The principles outlined in this article provide a strong foundation for developing innovative and customer-centric AI products. However, it's important to recognize that the AI space is constantly evolving, so maintaining a curious mindset and continuously asking questions will be crucial for ongoing growth and success in this field. Embrace the journey of exploration and keep pushing the boundaries of what AI can achieve.