Season 7 - Episode 5

Data Quality, Literacy and Discoverability by Avo CEO

In this episode, Stefania Olafsdottir, CEO at Avo walked us through her journey from math and philosophy to becoming a CEO. She uncovered the age of self-serve data governance, the key to democratizing data, and how to set up a data stack.

“Tell us about your story?”

The journey to being CEO is always straightforward. My background is in mathematics and philosophy. That’s what I studied. I was almost going into the hospitality business. I wanted to start a hostel in Reykjavik where bands would come and just rehearse music and prepare for Iceland airwaves. So I was super intrigued by that. I almost also just spent my life sort of just teaching kids. Then I stumbled into genetics. So I went to do a lot of data science stuff that distributed computing, analysis, of correlation between DNA mutation and physical traits like cancer and some new mutation or allergies and some mutation or being an alcoholic and some mutations, super interesting stuff.

That sort of meant that I was one of the very few individuals in Iceland, that back in like 2012 had worked with actual and real data. And not just like fabricated and fake data that you do when you’re studying for a Ph.D. or a master’s or a bachelor’s.

Then a mobile game blew up called QuizUp and reached 1 million users in their first five days. And they were looking for someone to join the team as a founding analyst, to understand why some of the people were going through an onboarding session and playing 10 games in one set and then coming, again and again, every day for like months. And then other people were just like, not even completing their email, and sort of understanding the difference between those groups and sort of how we might be able to help other people that were interested in downloading a game, but then they never got further.

Joining that company triggered both my sort of mechanical, technical brain doing computer stuff and calculating things and creating models and also my sort of philosophical brain and being very curious about people and sort of trying to understand what drives people? Why are products good? Why products like that for different people? That sort of was my trigger into the product analytics world, which eventually led to us internally at QuizUp building a lot of internal tools for sort of data, quality, data literacy, and data discoverability, which we now solve with Avo today. Avo is the analytics governance platform that allows product managers, developers, and data scientists to collaborate around updating their analytics for every single feature release.

“Back in the day, trying to get data you needed almost a Ph.D. because there was no easy solution. So as a product person or as a business person, how are you solving that problem with Avo today?”

I definitely think it’s a huge movement. It’s a wide shift that’s going on. It goes way beyond what we are currently building at Avo, but ultimately, the future is ours. Back in 2010, we started wanting to talk about things like self-serve analytics where people should actually be able to find information and find access to the questions that they had to answer because of their jobs. That conversation about self-serve analytics has really grown, but it hasn’t really fully been solved.

I was recently having a conversation about this on Twitter and stumbled upon a blog post that I thought also captured this well. I think we’re just talking about so many different things. When we’re talking about central analytics, we haven’t defined what it means to us. I think what it means to me is self-serve analytics is being able to access the metric that you need to be able to report on week over week or day over day because of your job. You are able to access that data on your own without help from anyone else. Self-serve analytics does not mean you should have access to any type of data at any given point in time, and just be able to make assumptions based on it.

I think we’re far away from that world. And I think we have to realize that that still requires two expertise to come together. It’s the domain expertise of the question that you’re trying to answer, and then it’s the data expertise of understanding the data sets that you’re working with and understanding just all of the different sorts of intricacies of working with huge data sets and how they might be misleading and all those things. Which just comes from the experience of working with data all the time, all day, every day.

“Where does your product sit in this whole world of analytics?”

To give you a little bit of a deeper background, and a backstory of the QuizUp adventure. So QuizUp eventually grew to have a hundred million users around the world, and it was backed by Sequoia and Tencent, but the team was entirely Icelandic. We were building this at a time when product analytics wasn’t even a concept. When I was Googling how to calculate retention, maybe I found one blog post, there was no Product School. We had to figure that out, we went through three key learning points and the data journey of QuizUp.

One was this sort of analytics thing. Nobody was able to answer any question without going through a data expert, because all of the data was really broken and nobody knew where to find it. It was just so scattered. So you had to be a data expert, no SQL, no Python, no Bash to be able to stitch together all of these different data sources. In two weeks you would be able to answer your question, counting, how many users did X. It just wasn’t a sustainable model.

We wanted to transition over to more of a self-serve analytics model. What that meant is we adopted a Mixpanel and Amplitude – actually, we had both of them. The idea was that product managers, developers, the CEO, marketing people, would be able to go into some UI to answer their basic questions.

We would still be working on some raw data to answer the more in-depth question and do some ad hoc analysis. That still required stitching together some data sets, but we would empower the team to go through their self-doubt analytics motions for things that we could. That was the first realization.

The second realization came around. We’re helping them do self-serve analytics, but they have broken fishing poles and have no hooks and just everything is broken. So they aren’t really fishing for anything from self-serve analytics. And that was because the data was still going in such a terrible way into the tools into Amplitude and Mixpanel. That was the birth of what later became Avo where we developed sort of rigorous planning processes and schema management, for how the analytics events should be instrumented before they would go into tools like Amplitude and Mixpanel.

We developed this concept called the purpose meeting, where we would have a sit down with a data scientist, a product manager, and I was a developer, a designer, and a back-end developer. We would kick a feature development process off with a purpose meeting where we would define, what is your goal with this feature? How do you want to measure its success? And then after that, we would define what are the data structures that we actually need in order to be able to measure these things?

The purpose meeting was a game-changer of a process for everyone because it just sparked everyone’s interest in wanting to understand how features were doing. It just changed the developer’s mindset from being like, I just want to get this feature out over to being like, ‘Hey what needle am I actually trying to move here?’ And being curious about the old features that they did, how are they doing? And sort of sparked this movement for people wanting to iterate on what they had already shipped because we wanted to measure the success of that. And we wanted to move some needles with it.

Avo is sort of that data analytics, planning, and schema validation tool that product managers and developers use so that they can actually successfully measure the releases of their products. Avo integrates them with the tools like Mixpanel, Amplitude, DBT, BigQuery, all of those data tools that you use so that you can port the schemas that you defined over to there. Then you also have validation that the teams are actually, they match what you defined.

Fast forward, maybe a couple of years later, we had just evolved this so much at QuizUp internally. First of all, we learned that we actually had created this world where we want to never release our products without proper analytics. And so every time we wanted to release a product, people were so frustrated because they were like, ‘The product is ready, but I haven’t done analytics and it’s not ready. I haven’t validated it.’ And so we created this new type of bottleneck. Previously the bottleneck was that decisions can’t be made because people don’t have access to data. Now, the bottleneck is we can’t release product updates because we haven’t validated our analytics.

And then sort of, we moved onto the next step, which is the third key milestone, this process of maintaining analytics, schemas, and metrics that also needs to be democratized. It definitely always needs to have an expert, someone who really understands data, but there can’t be a single gatekeeper. There’s so much discussion about like teams need to have, or companies need to have a data governor.

I agree. There’s a stage where they have to have a data governor, it’s the stage where you’re going from data chaos to some sort of a structure, the next step is you really need to decentralize that data governance. You’ll have to have, like an over the viewer, someone who can help you make sure you’re doing the right things, but everyone on the team should be able to understand what is good data. How do I define good data? How do I use good data? So it’s all about increasing data quality, data literacy, and data discoverability, so that’s the journey.

“Isn’t it amazing when you hear someone from the marketing team or support saying what is our success metric? How do we measure this?”

Oh my God, that feeling. I remember, when at first when we started developing this, we would always pound on the door of product teams. Hey, I heard you were releasing a feature, can we make sure I measure it? But then, once we had sort of done the purpose meeting a few times eventually, it warms my heart still to think about it. And I still get requests from some of my old coworkers to actually facilitate these meetings. It turned over to being like, ‘Hey data team, we’re about to release a feature, would you mind facilitating a purpose meeting for us?’ And then the next step was like, Hey, data team, we just did a purpose meeting. Here’s the output? Would you mind reviewing it for us?

That’s the beautiful maturity of centralized analytics over to self-serve dysfunctional analytics over to self-serve analytics. And self-serve data governance. That was a very rewarding journey. I totally relate to that feeling when other people in the team or in the company are just like that feeling. When I saw someone have a Mixpanel chart open, I was like, ‘What are you, what are you doing? What are you working on? Tell me more.’

“What are some good practices for product teams to set up their own data stack?”

Like with so many good questions, the answer is, it depends. The two key dependencies here are, number one, where are you now? I mean, culture-wise and tool-wise. Do you have any data? Do you have no data? Is there buy-in from leadership to make data-driven decisions or is everything based on gut or a three-year-old roadmap that nobody really wants to move?

The other, I think the other dependency is sort of, what is the skillset on your team? Do you have access to analytics, engineering? Do you have access to anyone who knows SQL? Do you have access to developers? Are you releasing product updates biweekly? And would the developer be up for making sure there’s analytics along with every single one of those feature releases? So all of these are dependencies that before I would recommend anything, I would always ask questions around this.

For example, for someone who has no product analytics in place currently, I would say start by choosing a product analytics tool and choosing a single north star metric for the product that you’re building.

Let’s say you’re migrating a newspaper over to being digital. You’re doing the digital transformation for a newspaper. And let’s say your north star should be something like subscribers who are reading articles every week. Then you have a few key inputs into that metric. I would always recommend identifying that sort of business metric that helps you make better decisions about your product. identify the inputs that actually drive that north star metric and then choose like two to five max. Initial events that you want to implement and just start looking at the data for that and feel how that starts triggering more questions from you. My recommendation for someone who has to stitch their data stack together is to start with a good question and go from there.

“I think asking questions is the life blood of a PMs. Asking questions, trying to get to more questions. ”

I often joke that the key responsibility of any good analyst is to dig deeper. When someone asks you a question, a good analyst will always answer a question with a question because people never ask you for literally what they want to know. They are trying to understand something they have in their minds mapped that understanding to seeing some sort of a metric. And then they just ask you about that metric. But you, as someone who’s a data expert, know there are multiple ways to think about the impact of that metric. And so the first question is always, what are you actually trying to understand?

The same thing applies to product management as does to sort of analytics and where should you start? It’s always about slicing. You have some exciting direction, you could take your product into, and there are rarely bad ideas that fall to your table. So your job is to say no to really good ideas all the time and just like deprioritizing really good stuff. And just arguing for like why you should not do that and why you should do this instead.

A really similar thing applies to analytics as well. Obviously, you have all of these questions that you want to answer with data, but number one, you won’t have time to get all of the data instruments that you don’t already have, and number two, you won’t have time to dig into all of the data. So you really need to prioritize.

What are the most pressing questions that you should start by answering and then work? It’s a cycle, you look at your data and you make decisions on your product. Data is never going to tell you what to build, but it can act as a guiding light along with your gut feeling and qualitative research.

“How do you apply that framework for yourself as a CEO? And what does your day-to-day look like?”

A lot of these big tech companies have started these training programs because the product manager role has just turned out to be such an important role in business success. Product is fundamental, product-market fit and identifying it, and fitting your customers to your product is such an important part of being a successful business today, particularly in the digital world of subscriptions, and all those things.

But I think what’s interesting is, I think Marty Cagan said, that there’s a high proportion of today’s CEOs that started off as product managers because it requires the same sort of pillars of thought.

It’s one you need to obsess about the usability of your product– that’s like the end-user. You need to obsess about the desirability of your product– that’s the market who wants to buy it. You need to obsess about the engineering feasibility of your product. Can we build it? Does it make sense? And then you need to obsess over the business liability of your product. Will someone pay for it? How much will that pay up? The engineering and development costs, the marketing costs, the customer acquisition costs, and all that stuff.

In the modern-day, the role of a startup CEO is very similar to the role of the product manager. These are the four things that I obsess over in my company in addition to, team and financials.

My job is to maintain a good mission, build a great team and make sure we don’t run out of money. That’s how I see my job.