How to Be a Great Machine Learning PM by Google Product Manager

In this talk, Ruben Lozano illustrates what Machine Learning is, how it differs from Artificial Intelligence and how it is related to statistics. He also explains that data is the foundation on which the machine learning algorithms work, and predict future outcomes. 

Further, he narrates that it is not advisable or practical to use ML in every scenario, and also on how to collaborate with data scientists to evaluate the training model, and use certain methods to clean and make sense of the data.

Ruben Lozano

Ruben Lozano is a Product M at Google Cloud, overseeing customer experience for all the online experiences before customers begin using the cloud. He also worked at Amazon for three years before joining Google, managing a team of senior product managers.

Machine Learning for Product Managers

What is Machine Learning (ML)?

Unlike in classical programming, in Machine Learning, you don’t have rules beforehand, but you do have a lot of data to make rules.

ML and Statistics

Supervised Learning

• Regression (Quantity) 

  • Linear 

  • Ridge

  • Lasso

• Classification (Category)

  • Trees

  • SVM 

  • KNN 

Unsupervised Learning

• K-Means 

• PCA 

• Collaborative Filtering 

To ML or Not to ML, that is the question!

Implementing ML begins with data. Once enough relevant data is obtained, it is used for predictions, and these predictions are in turn, used for better customer experience, which has the potential to bring more traffic. This traffic again leads to more data. Hence, a cycle is formed.

ML can be implemented when your problem…

• Handles very complex logic 

• Scales-up fast 

• Adapts in real-time 

• Requires specialized personalization

• Has existing examples of actual answers 

Sample ML Problems

• Ranking: Helping users find the most relevant thing

  • Example: Ranking algorithm within Amazon search

• Recommendation: Giving users the thing they may be most interested in

  • Example: Recommendations from Netflix

• Classification: Figuring out what kind of thing something is

  • Example: Product classification for Amazon catalog

• Regression:

  Predicting a numerical value of a thing

  • Example: Predicting sales for specific Amazon products

• Clustering: Putting similar things together

  • Example: Related news from Google search

• Anomaly: Finding uncommon things

  • Example: Fruit freshness

Ruben then shows us a chart depicting that ML can be implemented when your data…

ML should not be implemented when your problem…

• Can be solved by simple rules 

• Does not adapt to new data 

• Requires full interpretability 

• Requires 100% accuracy

And ML should not be implemented when your data…

What do you need for ML?

1. People

2. Processes

3. Tools and Systems

Formulate the problem

1. Problem

2. Data

3. Features

4. Model

5. Productionize

Great ML problems cannot be productionized due to high implementation costs or inability to be tested in practice.

Preprocessing the data

1. Cleaning

2. Sampling

3. Unintended bias

How do you evaluate the model?

How to best partner with scientists?

Treat your ML project as a partnership: 

“A PM from an ML project I worked on basically threw the requirements over the fence to me and was mostly unavailable. To meet timelines, I kept moving forward. Unfortunately, the deliverable at the end of the three-month project, though aligned with initial business requirements, was not what the PM wanted and didn’t meet the need. The model never made it into production and we really didn’t gain any learnings.” 

Have a clear problem, hypothesis, and success metric

“PMs who come prepared with a clear, preferably data-driven, problem and hypothesis will have a much more productive discussion with me than otherwise. The problem definition need not be perfect, but I do want to understand what’s been tried, why it isn’t working and what we’re aiming for.” 

Be willing to make tradeoffs 

• Time vs Quality

• White Box vs Black Box

• False Positives vs False Negatives

• Go vs No-Go Metrics 

• Help get data and explain it

• Scientists are not Software Engineers

• ML creates tech debt

• Be considerate of scientist time and momentum 

Ruben concludes by saying that there’s always room for experimentation, especially in big companies. Only with more experimentation can the Product Managers go deeper and make the most use of machine learning.