16 Key Data Skills for PMs in the AI Era (Second is Crucial)

Data skills for PMs are the abilities that let product managers turn product, customer, and model data into decisions. In AI products, data shapes what the product does, so a PM's data skills now shape the product itself.

Of the 12 adoption and scaling practices McKinsey tested for its State of AI report published in March 2025, tracking well-defined KPIs for gen AI solutions had the biggest effect on the bottom line. Yet, fewer than one in five respondents said their organizations were doing it. I read that as a data skills gap, and it sits squarely with product managers.

This guide ranks the data skills that matter most for AI-era PMs, shows how each one changes when the product has a model inside it, and gives you a concrete way to start on each. For the foundations, including the types of data PMs work with and how analysis fits the product workflow, start with our guide to data analysis for product managers. This piece picks up from there.

Key Takeaways

• Data skills for PMs are the abilities to define success in measurable terms, judge whether evidence can be trusted, and turn that evidence into product decisions.

• In AI product management, data shapes how the product behaves, which turns data skills from reporting work into product-building work.

• The eight core data skills for AI PMs, in priority order, are metric design, AI output evaluation, data quality judgment, statistical thinking, experiment design, structuring unstructured data, self-serve querying, and turning analysis into decisions.

• Defining the right metrics ranks first because every eval, experiment, and dashboard inherits that definition of success.

• AI evaluation is continuous: production failures should feed back into the test set, because AI systems keep changing after launch.

• SQL still matters, but AI has shifted the scarce skill from writing queries to verifying them.

• PMs need enough statistical and evaluation literacy to own the question, the threshold, and the call, while analysts and data scientists own the methodological depth.

What Are Data Skills for PMs?

Data skills for PMs are the abilities a product manager uses to define success in measurable terms and turn it into product decisions. Analysts and data scientists own the depth: pipelines, models, and statistical methods. Product managers own the question, the definition of success, and the call that comes out of the data.

Data skills are one part of the broader set of product manager skills, and they overlap with technical fluency. What changed is the scope. On AI products, they now include evaluating model outputs, judging the data a model learns from or retrieves, and deciding when a probabilistic system is good enough to ship.

Why Data Skills Matter More in AI Product Management

AI changes the job data inside a product. In a traditional feature, data describes what happened. In an AI feature, data also decides what happens next, through the examples a model learned from, the documents it retrieves, the instructions that configure it, and the AI eval sets that gate its releases.

Three shifts follow from that:

• Data shapes behavior: A stale document in a retrieval index becomes a confident wrong answer, and a skewed dataset becomes a skewed feature.

• Quality becomes a distribution: The same input can produce different outputs, so quality has to be measured across many examples instead of being verified once.

• The system keeps moving: Models, prompts, and user behavior change after launch, so evaluation and monitoring become continuous work.

Here is how each core skill changes when the product has a model inside it:

The 8 Core Data Skills for PMs, Ranked for AI Product Work

I ranked these by how much a gap in each one costs a team when the product has a model inside it. SQL sits at number seven because AI has made it the easiest of the eight to borrow, while the skills at the top are the ones no tool can supply for you.

1. Defining the metrics that matter

Metric design is translating a product goal into a small set of measures. This could be a one outcome metric that proves value, a few leading indicators that move first, and guardrails that catch harm. It ranks first because every eval, product experiment, and dashboard inherits the definition of success you set here.

AI features make this harder because activity can look healthy while value stays flat. In an AI assistant, more queries per user can mean people are rephrasing because the first answer failed. Successful sessions, where the user got an answer and acted on it without asking again, tell you far more. 

I'd also put cost per successful task on the scorecard from day one, since a feature that succeeds expensively can still fail the business. We unpacked the organizational version of this trap in why AI usage isn't translating into value.

Kunal Mishra, Group Product Manager at Amazon, recommends defining success across four buckets before any code is written: business, user, technical, and risk mitigation. His rule from a Product School webinar is simple: "If it's not on paper, it doesn't exist."

Start here: Write your AI feature's success definition across those four buckets and add one guardrail. Our list of 20+ key metrics for product management covers the classics, and our guide to outputs vs. outcomes keeps the two apart.

2. Evaluating AI output quality

Evals show you the whole distribution of quality, including the tail where users get hurt.

The core vocabulary is small. Precision tells you how often the system is right when it acts, and recall tells you how much of what it should catch it actually catches. Favoring one over the other is a product decision about which mistake hurts users more. 

Add groundedness and hallucination rate for generative features, and know when an LLM judge is reliable enough to grade outputs and when a human has to. Our guides to AI evals for product managers and AI evaluation metrics go deeper.

What most teams underestimate is that evaluation never finishes. Ashley Nutter, SVP of Product at CNN, described the loop behind CNN's AI system at ProductCon New York:

"Ultimately, you want to make sure that you are evaluating the system and that those inputs are feeding back into the system so that it gets smarter over time, and also so that it catches any new issues that it introduces to itself, because the system is dynamic."

Watch

In practice, every production failure becomes a new test case, and every prompt or model change runs against the full set before it ships. We broke down CNN's architecture into the one architecture decision that fixes failing AI agents.

Start here: Pull a few dozen real inputs for one AI feature, including the messy ones, label what good looks like with a teammate, and rerun the set after every change. Product School's AI Evals Certification covers the full loop, from failure-mode discovery to calibrating LLM judges against human labels and feeding production traces back into testing.

AI Evals Certification

Learn to build trusted AI products. Design eval suites, integrate CI/CD gates, monitor drift and bias, and lead responsible AI adoption.

Enroll now

3. Judging data quality and AI readiness

Data quality judgment is knowing whether data can be trusted for the decision or model it feeds. It climbs the list of AI products because bad data produces wrong behavior at scale. 

In February 2025, Gartner predicted that through 2026, organizations would abandon 60% of AI projects that lack AI-ready data, and reported that 63% of organizations either lack or are unsure they have the right data management practices for AI. Gartner's bar is useful for PMs too: AI-ready data has to represent the specific use case, including its patterns, errors, and outliers.

PMs can own this without building a pipeline. Before an AI feature ships, ask five questions:

• Source: Where does this data come from, and who owns it?

• Freshness: How old is it, and how often is it updated?

• Coverage: Which users, languages, or edge cases are missing?

• Labels: Who labeled it, and how often do labelers agree?

• Permission: Are we allowed to use it this way?

For retrieval-based products, ask the same questions about the documents behind every answer, as our guide to RAG for product managers explains.

Start here: Run the five questions against the data behind your most important AI feature. Every question you can't answer goes on your data readiness backlog.

4. Thinking statistically

Statistical thinking means reading data as distributions rather than single numbers. It’s knowing when a change is real or noise.

AI makes this essential. The same prompt can produce different outputs on different runs, so one test run tells you little. A 90% pass rate can hide an assistant that fails most refund questions in one language, and those users churn quietly long before the average moves. Three habits cover most of the ground:

• Segment by default: Break results down by language, query type, plan, and new versus returning users before trusting a total.

• Repeat what matters: Run important eval cases several times and report the range alongside the average.

• Check significance before celebrating: A lift inside the normal noise band is a hypothesis to test again.

Our guide to AI experimentation covers how to handle that variability in practice.

Start here: Break your team's headline metric down by three segments. If one moves against the average, one has found their next question.

5. Designing experiments that hold up

Experiment design is testing a change so the result can be trusted. It includes a clear hypothesis, one primary metric, guardrails, a large enough sample, and a stopping rule agreed before launch.

On AI products, every prompt change, model swap, or retrieval tweak is a product change that deserves the same rigor as a new checkout flow, and AI adds steps before the A/B test. Offline evals tell you whether a change is safe to expose, shadow mode compares new outputs with the current system without users seeing them, and a canary release limits the blast radius. 

Run AI tests long enough for launch curiosity to fade, since early usage spikes can flatter a feature.

Our product experimentation playbook for AI PMs walks through eight experiment types and when to use each. Product School's Product Analytics & Experimentation Certification goes deeper, including a full module on methods for when a clean A/B test isn't possible.

Blog CTA: Product Analytics & Experimentation Certification

Pair analytics with trustworthy experiments, and use AI to surface the insights that sharpen your decisions.

Enroll now

Start here: Before your next test, write the hypothesis as "We believe X will change Y for Z because...," name one guardrail, and agree on the stopping rule up front.

6. Turning unstructured data into structured data

In other words, using AI to turn transcripts, support tickets, sales calls, reviews, and video into structured fields a team can count, filter, and query. At its Data & Analytics Summit in London in May 2026, Gartner put unstructured data, such as documents and multimedia files, at 70% to 90% of organizational data. That's where most product knowledge lives.

The skill sits in two places: designing the schema and checking the extraction. Choose fields that would change a decision, such as topic, intent, severity, or product area, then measure how often the model gets each one wrong. 

Different fields deserve different bars. In our breakdown of the CNN system Ashley Nutter described at ProductCon New York, a cheaper transcript-only pass identified three of the four Artemis crew members in a clip. Only the more expensive visual pass, which read the name on a flight patch, caught the fourth. 

A transcript is good enough for finding soundbites, while identifying everyone on screen justifies the expensive pass.

For customer feedback, the rule we teach in our Claude Code for PMs Certification is to "ask for a count, a grouping, and a quote — never a summary." Counts and quotes can be checked against the source. Our guide to customer sentiment analysis covers the basics.

Start here: Export last month's support tickets, define five fields that would change a roadmap decision, extract them with an AI tool, and hand-check a sample before trusting the counts.

Claude Code for Product Managers

Claude Code is already changing how the best PMs work — at Stripe, Linear, and the fastest-moving teams in tech. Use the AI tool reshaping product work to prototype, analyze, and ship, before a ticket is written.

ENROLL NOW

7. Querying data yourself

Querying data yourself means answering first-order product questions without filing a ticket, through SQL, a BI tool, or an AI assistant, and knowing enough to check the answer. It ranks seventh because AI has made writing queries cheap, and checking them is now the scarce part.

AI-written SQL can run cleanly and still be wrong. A join that duplicates rows inflates revenue, a missing filter counts internal test accounts, and a timezone mismatch moves a launch-day spike. Learn enough to read a query and spot those mistakes: SELECT, WHERE, JOIN, GROUP BY, and basic window functions, plus a working map of your own tables and events.

As we covered in Claude Code for product managers, PMs can now query their actual data in plain English while the context is fresh. That's exactly why our Claude Code for PMs Certification dedicates a module to verifying product metrics: looking at the rows behind a number and working it out a second way. I'd call it a core habit for any AI Builder.

Start here: Take three questions you'd normally send to an analyst this week. Answer them yourself with AI assistance, then ask the analyst to review one query.

8. Turning analysis into decisions

This is converting evidence into a call: leading with the recommendation, showing the data that supports or challenges it, stating your confidence, and naming what would change your mind.

AI products raise the bar because the evidence is probabilistic. Stakeholders tend to hear "92% accurate" as "it works," so the PM has to explain what happens in the other 8%, who it affects, and whether that's acceptable for the use case. That means agreeing on ship, hold, and rollback thresholds before results come in. It's also what senior PM interviews increasingly probe, as we covered in our guide to senior PM technical interviews.

A one-page decision memo keeps you honest: the decision needed, your recommendation, the evidence, the risks, your confidence, and the trigger for revisiting the call.

Start here: Rewrite your last metrics update so the first line is a recommendation and the last line is the condition that would reverse it.

8 Helpful Data Skills That Round Out Your Toolkit

These eight rank lower because a teammate or a tool can cover them more easily than the core eight. They still make you faster and more credible as an AI product scales.

A Data Skills Self-Check for PMs

Answer each question with a yes or a no. Every no points to the skill worth working on next.

• Can you name the outcome metric your AI feature exists to move, and its guardrail? (Skill 1)

• Do you know your feature's current eval pass rate and what changed it last? (Skill 2)

• Do you know where your feature's data comes from, how fresh it is, and who owns it? (Skill 3)

• Do you know which user segment your AI feature performs worst for? (Skill 4)

• Did your last experiment have a guardrail and a stopping rule agreed before launch? (Skill 5)

• Could you turn last month's support tickets into counts you'd defend in a roadmap review? (Skill 6)

• Could you answer a product question yourself this week and verify the query behind it? (Skill 7)

• Does your last data update open with a recommendation and a confidence level? (Skill 8)

• Can you state your AI feature's cost per successful task? (Skill 10)

If you answered no to three or more, start with the highest-ranked skill among your no answers. The top of the list does the most damage when it's missing.

Data Skills Are Product Skills Now

The PMs who stand out on AI products define success precisely, know when an AI output is good enough to ship, and can tell whether the data underneath a feature deserves trust. Those are product judgments expressed through data, and they compound: every metric you define well makes the next eval, experiment, and decision easier.

Start with the first two skills on the list, because everything else inherits them, then build outward as your product demands.

If you want structured practice, Product School's Product Analytics & Experimentation Certification builds data literacy and experiment design through hands-on labs, and the AI Evals Certification teaches you to build calibrated eval suites and close the loop from production back into testing. If you're newer to AI product work, I'd start with the AI Product Management Certification.

Product Management Foundations Certification

Use the most popular certification to learn the core skills, mindset, and tools to become a great Product Manager in the age of AI, and ship products people actually use.

Enroll now