Leveraging A/B Testing Data with Pivotal Software Sr PM

Last week, Product School hosted Pia Chakrabarti, a Senior Product Manager at Pivotal Software for an exclusive #AskMeAnything session. Pia lent her expertise in A/B Testing and data analytics to discuss measuring success when conducting research. She also provided advice on transitioning to product management for engineers and the PM-curious.

Meet Pia Chakrabarti

Pia a data-driven Product Manager aiming to delight users by building intuitive, impactful products, and believes that the best products are made by strong teamwork. Her expertise includes analyzing qualitative & quantitative user feedback, A/B testing & experimentation, user research & user-centered design, product strategy & roadmap planning, defining success metrics & exceeding targets, managing stakeholders, and collaborating across teams. Currently, Pia works as a Senior Product Manager at Pivotal Software.

In your experience, how important is digital-first strategy in the product domain? What is your experience with implementing digital-first strategies?

In this day and age, a digital-first strategy is almost mandatory! For better or for worse, we spend most of our days tied to a device and so if brands are not taking a digital-first approach, chances are they are going unnoticed. In my time as a consultant, I have worked with a few clients who were facing challenges migrating to a new way of life.

For example, an entertainment company comes to mind – they were so accustomed to booking gigs through word of mouth and it was difficult for them to change their mindset. However, after working with us they were able to see the value of moving to digital and the numbers spoke loudly as well!

How do you identify areas where A/B testing can add value?

You need large amounts of traffic to see results in a timely fashion. For this reason, A/B tests are particularly valuable to run on landing pages, pricing plan pages, etc. You can use a sample size calculator to figure out how much traffic you will need/how long you will need to run a test using a sample size calculator such as this one: Sample-size calculator.

How do you determine what is a viable ‘B’ in your test?

First, I’d like to point out that there can be a B, C, D, etc. and there generally are several different variations. Choosing the variations to test against control comes down to using the same PM decision making skill that you use otherwise day-to-day.

You identify the biggest problem, ideate solutions, then prioritize those solutions based on some criteria. The top solutions are those that should be tested as variations against the control in an A/B test. The PIE framework is useful for prioritizing your A/B tests.

work from home

How do you execute your test in a production environment where people expect an application to not have constant change short of value-add releases?

People sometimes feel nervous about doing A/B tests because they don’t want to compromise the user’s experience. If you feel this way you should consider that perhaps your original experience (A) is much worse than the experience one of the variations of the A/B test could provide. In that case, isn’t it smarter to run the test and let the results speak for itself? You will always have access to live data and can turn off the test at any point if numbers do not match your expectations.

How do you measure the success of your tests and outline a go-forward plan that leverages what you’ve learned from your A/B test?

Keep the team and stakeholders in the loop about the tests you are running and share results often. Also, include them in the ideation process. This will keep everyone informed and motivated, eager to make product decisions when the data shows it’s time.

I’m PM-curious and interested in moving into a PM team role with transferable skills but not necessarily deep product experience.

The best way to move into a PM role would be to move internally within your company. Ask PMs in leadership roles (director etc) if you can start to shadow, then start contributing, then hopefully when the next role opens up it will be yours.

If you are unable to move within your company, I would recommend working on a side project so that you have something real to show at interviews. Hackathons would be a great way to do this as well. Another thing to consider Junior and Associate PM roles.

You may also be interested in: Everything You Need to Know About APM Programs

Could you talk a bit about your technical expertise and how well you collaborate with engineering when you may not have as much coding knowledge as them?

This is an extremely relevant question for me in particular because up until now I have PMed fairly nontechnical products. The products I worked on at the last company were all consumer-facing, in the career-building space (resume writing, interview skills, etc.). The company I currently work for (Pivotal, now acquired by VMware) is in the B2B cloud infrastructure space. I had little knowledge about this space before joining the company and the dynamic between PMs and engineers is definitely different.

I find myself using a lot more facilitation skills and working with my team constantly to remain aligned on the roadmap, current status, next steps, etc. I also often leverage the expertise of the engineering lead in conversations with stakeholders and customers. We call this method of working a balanced team mentality.

We are all responsible for the success of our product and collaborate heavily to make sure we are doing the right thing. So in conclusion – I do think it’s possible for PMs to add value to a technical product without being very technical if the team is supportive of a balanced team model.

Have you ever been a part of creating a vision for a digital product? If yes, can you describe the process? 

A vision of any product should be directly aligned to the company vision. So I would start there – why does this particular company exist? How is this company changing the lives of its customers? I would talk to leadership, different teams such as sales and customer service, etc to get a better understanding of all this. Based on what I learn, I would set up a workshop for my team to collaborate on the vision – this is not something that can be done alone by the PM. Ideally, you’d want a facilitator from outside of the team to run the workshop so that all voices are heard equally.

What are the first steps to focus on when making the move from Software Engineer to Product Manager?

It seems to me you already have many of the PM skills under your tool belt. It will be a matter of telling your story like a PM. I’d spend some time looking through PM job listings – select 3 that speaks to you. Of those 3, highlight the skills that they have in common. You will most likely see overlap and this will indicate what kind of skills companies expect PMs to have.

Start writing out your big accomplishments throughout your career and compare against those highlighted skills. What have you done that is similar (and may need reframing)? What have you not done yet? The things you have not done, you could start to do now.

Additionally, I would recommend you to apply to jobs within an industry you have worked in as industry knowledge is always valuable as a PM. Of course, when I say “apply” what I really mean is to network and apply through a referral. Cold applying is not worth it. I’ve seen many Software Engs/Project Managers/and CEOs transition to PM roles successfully.

What do you look for in a Product Management resume and/or candidate?

First and foremost I look for strong communication skills and organizational skills. Technical skills are always a plus!

Did you miss this event? Check out our events page to sign up for the next #AskMeAnything session!

Enjoyed the article? You may like this too: