Machine Learning Meets Product Management by Microsoft Product Manager

This week, Product School hosted Alok Jain, Product Manager at Microsoft, for an #AskMeAnything session. He explores a wide range of topics such as machine learning, metrics, skills, knowledge, and advice to aspiring PMs!

Meet Alok Jain

Alok is a Product Manager in the Machine Learning heavy domain. He has an insatiable thirst for knowledge and new experiences, passionate about technology and its positive impacts, thoroughly enjoys extracting actionable insights from big data and building amazing Products that lead to customer satisfaction. Currently, Alokik is a Product Manager at Microsoft, thought having worked various roles with different responsibilities while there.

Advice for Aspiring PMS: Skills, Metrics, Transition, and More!

If you were an interviewer or hiring manager, could you describe what traits and skills you would be looking for in candidates without any experience? Could you also advise beginners on how to get their first job in product management?

This is a great question. Today everyone has a relatively similar background when applying for PM roles. So CVs start looking very similar at a junior level. I would specifically like people who have shown a real interest in the domain of product management. Just reading books or doing a course may not make the cut.

This would mean having done some internships in the space, built your own products, doing some user research, and trying to improve and iterate on your product. I also like folks who have worked in roles that are similar or assisting the PM roles, for example, some business analysis, data analysis skills, or have worked or interned on a customer relationship role.

Knowledge

What is your advice to people trying to crack the APM program?

APM programs today have become very competitive. Many of you would have got a lot of advice from many people. I would like to share my journey. I wanted to be a PM for the longest time but back in the day ten years ago there were fewer opportunities in my home country, India.

I found it really hard to crack any interviews as I was always filtered out at the resume review process. I then decided that I would do supporting roles to the PM role and slowly build myself up to be a successful PM someday. So I was a developer for a year, I was a data analyst for a year, a business analyst for another year before I got my first break. However, when I look back now I think I am very glad that the path to PM was circuitous for me.

I gained a lot of hard skills along the way that differentiates me from my peers and helped me stand out. While I may not have answered you question the way you would have liked what I am trying to provide here is my perspective that there are more ways to land a PM role if you control the time horizon and navigate your way through.

I am good at one to one conversations with business users and stakeholders, but I am finding my user interviewing skill not up to the mark while in a group teleconference, how do I improve this?

I would recommend doing online courses on user research and qualitative research to improve these skills. At the end of the day, good conversations come down to a drive towards clarity. If you are speaking to dev stakeholders ask yourself if you are speaking to them in their language.

Fluff talk with devs pisses the devs off. Similarly, if you are doing user research asking leading questions will give you answers that you want to hear and not really help you in building out the product vision. Similarly when you are with your design team trying to tell them what to do (eg. make pixels look like this) can lead to disastrous conversations.

Conversation

What are the criteria you use to define success metrics? What percentage would you advise an upcoming PM to start with?

There are 2 types of metrics for any product.

  1. Business metrics: Monthly Active Users, Daily Active Users, Monthly Engaged Users, and Daily Engaged Users. This set of metrics is very important for the senior leadership (VP/CPO/CEO + finance to track) This would determine to fund for your workstream. As a PM you care for this number to move up.
  2. Product truth metrics. This allows a PM to truly add value. These metrics define if you are able to deliver true user value. If you build a product that helps you get to your physical meetings on time your product truth metric would be % of users who got to the meeting on time given the intervention of your feature. Is that number better than without having any intervention in the first place? As a PM you need to move the needle on your product truth metrics.

Interested in metrics? Check out: These Are the Metrics Great Product Managers Track

What is the pathway for non-technical, business, or marketing graduates to land a product job?

Read up all the PM-related books, blogs, follow as many PMs as you can on Linkedin. Take a course on data analysis. Build small projects or features for small use cases that you observe as daily pain points, it could be a problem that you, your family, your friends face.

Try getting more people hooked on to it. Watch a bunch of systems engineering and architecture videos. Youtube has plenty of great channels. Use products that you are interested in working on and reach out to PMs who manage those products and provide them a detailed view of what is lacking and what could be made better. Not an exhaustive answer but a good place to start.

You might also be interested in: Product Management Certifications

All you Need to Know About Machine Learning

Have you always been working on Machine Learning (ML) heavy products?

The philosophy behind ML products is not to ship the coolest tech but to root the features in real user pain points. We built this feature called Play on the email on outlook because we saw a real need where users were trying to reply to emails while they were driving to the office. So we built an experience where a user could communicate hands-free with the outlook and teams apps to get caught up or to reach someone in the office.

The point I am trying to make here is that you start with a user problem or need. You validate that need through data and user research. You build out an experience that addresses this problem – either through intelligent ML experience or through simple UX flows and finally release your product and get feedback.

When should a product manager decide if a problem should be solved using ML?

I think ML is only required when you have found a user need or a pain point that needs to be addressed. Building cool tech because you can is not going to lead to great outcomes. Understanding real user needs and using ML as one solution to solving for the identified user needs is the way I would approach this.

What advice would you give a data scientist looking to move into an AI/ML PM role in the next 3-5 years? Ideally what types of projects should I attach myself to?

The biggest lesson learned from transitioning from a data role to a PM role is to believe that everything can be solved by hard telemetry data. Its good to be data driven but in many situations or scenarios there is no real data to build a product hypothesis from. This could happen in v0 products where you are still trying to find product market fit.

The lesson learned for me was to rely on qualitative data, research from a small sample audience even though its not statistically significant.  The idea here is to learn from the lived experience of the user, take a leap of faith with a process or experience improvement that might not have a lot of hard data backing and build something of value.

Once you find product-market fit and enter the growth phase you then start looking at data and A/B experimentation at scale and adding ML-based intelligence to your products if applicable.

Machine Learning

What advice would you give to an aspiring product manager trying to transition into Machine Learning projects?

As a PM you dont bring expertise in ML algos, you dont bring expertise in engineering systems. What you bring to the table is the user experience and how the ML systems are inadequate to solve user pain. If you are building a recommendation system you as the PM would look through the data to see how bad the recommendation were, what categories did the experience particularly deteriorate over.

Do a lot of data mining and analysis to convince your dev and ML partners. Talk to real users of your product and get qualitative feedback on the situations where the algorithms did not work. You identify improvement areas and set KPI goals (Eg Precision should be X% in this quarter. Recall should be Y% in this quarter, NDCG should meet a Z threshold.) You then groom a backlog of initiatives that get you to the targets. And you keep repeating this.

As a guy from the technical community, what is that one basic thing we as developers/data scientists might be missing or overlooking when developing the AI/ML products?

Getting caught up on a solution without fully understanding the problem. Products are built for a population. You are a sample of that population. Understanding that you have unique tastes and preferences and hence what works for you may not work for everyone else is a bit understated.

When you build, you build for a target audience (the average user of that audience may be very different from you). Really internalizing this is hard and won’t happen until you actually get to building something and see the feedback flow in.

Product at Microsoft: Tips, Success, and Experiments

Do you have any tips on getting hired or at least get an interview at Microsoft?

Microsoft is a very large company, this means PM roles are very different and depend on your organization and the business you are in. A PM in Bing is expected to have very heavy tech chops. A PM in Teams is expected to have great UX skills and design thinking. 

A PM in Azure needs to have great systems skills. My take is to first understand what exact role you think you are best suited to, then look up jobs in organizations that have those roles. This would mean going through the JDs on the Microsoft careers site and only applying to roles that make sense to you where you can add significant value.

research

How is success measured at Microsoft versus other similar products of companies? If you could describe the roadmap and how much you target each iteration?

This is a great question. Success is defined by the business outcome that you land. As the PM you are in a unique role where you define your success criteria/KPI for your feature. You are responsible for hitting those KPIs. You can choose to do a set of things to achieve those KPIs.

Ultimately you are in a position where you can self evaluate whether you were successful or not if you defined the right KPIs and drove the right business outcomes for your customers. As a PM you won’t be evaluated on how many features you shipped or how much time you put in. You will be rewarded for the results.

You might also be interested in The Key to Product Success by Uber Product Lead

 I’m curious to hear about how Microsoft goes about preparing and conducting product experiments and how they use those to guide their product roadmaps. Could you share any of your experiences on that process?

It depends on the team that you are in. In Bing Ads, the experiments are very data-driven. We mine data and figure out pockets of opportunity. However, in more UX heavy roles the experimentation is determined by a combination of telemetry and user research.

The PMs then come up with product hypotheses and build a few experiments and test them on an internal audience. Once we are on track to match or exceed the target KPI we then move the features to a larger user base to get feedback at scale. Finally, if the KPI targets are met we ship the product to a worldwide audience.

Do you have any final closing notes for this AMA?

  1. You should consult with a lot of experienced folks before making a decision. I think the path to PM can be hard to crack through standard APM roles because of the demand-supply dynamics. A better way to enter this industry is by getting relevant experience in supporting roles if you are flexible on the time horizon. Starting out in a smaller company would also bring in a lot of learnings that a generic APM program may not provide.
  2. ML features are always good to have not necessarily a must-have. Until you find the right user problem an ML solution is just cool tech.
  3. Today there is too much competition for product roles. Even if you are not a PM in the title but you act as a PM in whatever role you are in – you will see significant success later on. As a PM you own the customer pain and your job is to reduce it. Do it without being a PM in the title and you will have a simpler transition into the role.
winner

For more insights on Product Management, join us for our next #AskMeAnything session!

featured speaker

Enjoyed the article? You may like this too: