This week our #AskMeAnything session welcomed George Jeng, Product Manager at VICE, to talk about Machine Learning!
Meet George Jeng
George Jeng oversees growth and analytics initiatives at VICE Media. He’s a technology enthusiast that loves problem-solving and has a great day when he’s helped a person or company achieve a breakthrough in overcoming a challenge.
Some of his specialties include:
– Semantic Analysis & Machine Learning
– Recommendation Systems
– Optimization: MVT and A/B/n Testing
– Business Intelligence and Analytics
– Lean Startup
Media Machine Learning at VICE
How do you handle ML in media?
In Media, the biggest opportunity for Machine Learning (ML) is to automate the categorization of the enormous library of content that is published.
For content recommendations at VICE, do you use content-based filtering and collaborative filtering using ML models?
When it comes to ML, there are a lot of misconceptions of what ML can and can not do. So the first thing is to understand the capabilities and limitations of ML. Most people think of ML as a silver bullet, but when they realize the amount of training data that is needed, the scope changes.
At VICE, we use a combination of third-party classifiers that we then use to estimate the proximity and relatedness of the article.
We did a write up of it here.
How do you balance manual curation and Machine Learning to maintain a certain level of curatorship that consistently adds value?
Machine Learning will never replace editors. The editors are the voice of the brand and the heart and soul of any publisher. But ML can make their lives significantly easier by recommending how content should be tagged to streamline curation.
How do you balance tactics that increase engagement (e.g. content recommendations) with filter bubbles?
This is a great question, and this really depends on where you are building a product. At VICE, a good majority of our users come in through SEO or Social, so the engagement past the initial article is very low. This is a problem all publishers have.
So by leveraging content recommendations, we surface more relevant content to people to hopes we can provide them with a better user experience. But the likelihood that those users are coming back every day to read VICE content isn’t a problem I have
How can you make a recommendation tool accurate?
In terms of making a recommendation tool better, this requires constant A/B testing to see what combination of parameters drive the most lift.
Is there a product team at VICE for developing internal editorial tools, like CMS, DAM, some sort of storytelling tools? If yes, is it located in the newsroom?
Yes, we have a product team that is dedicated to working on the CMS and API that powers our digital portfolio. We work closely with the editorial teams, but we aren’t located in the newsroom.
Do you work directly with data scientists? If so, have you come across best practices and/or challenges in working with them?
We have a team of data scientists at VICE. When it comes to working with anyone technical (data scientists, engineers, etc.), I always like to take a step back and give them the business perspective of why we are doing what we are doing. This way they know why their work matters. This way it’s easier to focus on the things that matter.
Product Management Skills
How do you go about testing/finding out/validating if something is worth pursuing product wise?
There are 2 ways that I usually go about finding out if a problem is worth solving.
1. Go talk to people, talk to people that you think would have this problem and get their take on what they perceive the problem is. Make sure you are addressing a problem that is really there and not just one that you think is there.
2. After you’ve talked to a few people, create a lightweight MVP, wireframes, a simple landing page, prototype, something low touch, and see how people react to it.
At the end of the day, you want to constantly make sure the juice is worth the squeeze.
What software and tools do you use to manage the product development process? Specifically, what do you use to track the productivity/efficiency of your dev team?
At VICE, we use JIRA to manage our workloads, but we are pretty lean and everyone carries their weight so it’s pretty clear when someone isn’t being productive.
Do you have any recommendations for integrating a content team and a tech team?
I think it starts with open communication, and just having a safe forum where both teams can openly discuss what’s going on both sides of the fence. At Time Out and VICE, the tech and content organizations always worked tightly together.
What’s your top tip when drafting an API?
When drafting an API, I would highly recommend playing with a lot of other peoples API’s first. This will give you a baseline understanding of what people expect. Postman is a great tool for kicking the tires of an API.
I’m little struggling with our go to market plan. There is a disconnect between the marketing and product teams. Any advice?
This really depends on the maturity of the company and the product. Ultimately product and marketing need to understand the product market fit, or product will be making the wrong thing and marketing will be spending money that isn’t going to drive conversion.
How do you normally get customer responses on prototypes? Do you set up customer interviews or use something like usertesting.com?
This really depends on the product. If you are B2B, then you can find out through talking to other companies and getting feedback at conferences and meetups.
If it’s B2C company, depending on how widespread the problem is, you could go the usertesting.com route, or just join a few Facebook groups that you think hit the demographic you are trying to hit.
Have you found that adopting a single OKR system
I don’t think there is a one size fits all, and it also depends on how independent the team is. Generally speaking, everyone should know the party line, and where the org is going. What tools/process you use to do that is up to the team.
Is there any use of analytics in your product? How do you decide to roll out a new feature or pick one from backlog?
Deciding what to work on next is driven by our OKR process. We know what KPIs we need to improve, so deciding if it’s backlog item or a new feature depends on what we know at the time (usually driven by analytics/ab testing) on what will move the KPI more.
Intro to Product Management
What major problems have you face while working as a PM and how did you solve them?
When I started at VICE, one of the long term goals we wanted to do was to implement our content recommendation system. But there was a lot of foundational work that needed to be done to our API infrastructure, Content Tagging, and Understanding of how user’s interacted with the site.
Each line of work took several months to complete to get us in a place where we could have our own recommendation system.
Can you give some tips for when, in any product stage, things are FUBAR?
When things go FUBAR, Its usually a sign that you need more project management support. I generally follow the rule, “I want good news fast, I want bad news faster.”
Also thought this article is relevant.
What are some of your favorite A/B testing tools?
I’ve used Google Optimize and Adobe Target. At the end of the day to be good at testing, I ended up coding my tests in JS, which rendered the platforms moot. I just used the platforms to calculate statistical significance. Google Optimize has a free tier which is appealing.
How do you balance management requests for shiny new toys vs. feature enhancements that the team feels positive will drive actual conversion?
I find the best way to prove out the value of a few features is to try to build a small prototype that can be executed as an A/B Test. Hackathons are also a nice way to get a quick demo together.
Once you have something you can play with, you can get it in front of people to see if it would really convert as you had expected.
How do use Lean Startup principles in your daily work?
We try to validate our assumptions through A/B Testing where possible. So we’ll build a lightweight MVP before building the full blown feature.
What’s your best advice for answering case studies at the beginning of the interview process?
Ask lots of questions and check your assumptions before starting the case.
Do you have tips about how to start the culture of measurement inside the company using success metrics?
Try looking into OKRs. We set OKRs every quarter and that guides our underlying KPIs.
Do you have any final advice for aspiring Product Managers?
Last word of advice is to never stop learning and always stay curious.