AI Evals Certification
Define what good looks like for your AI. Catch failures before your users do, so even its non-deterministic outputs earn your trust.
Next Cohort
Starts in 22 days
Dates & Schedule
Nov 7 – Nov 21
Sat, 7:30 AM–12:00 PM PT
Instructor
Jaswanthi MeganathanAI Product Leader at Google
Learn live online
Taught by top AI product leaders
Small class sizes
20 students per class, on average
4.8
Based on 1,284 ratings
20,901+
students graduated
Our instructors work at
The new standard for AI product reliability
As AI adoption accelerates, PMs face a new challenge: proving not just what AI can do, but whether it can be trusted. Customers, regulators, and leadership are demanding AI systems that are accurate, safe, and aligned with company values—but most teams lack the tools to measure and enforce trust at scale.
The AI Evals Certification equips Product Managers with the frameworks and playbooks to lead in this new era. You’ll learn how to define “good” beyond accuracy, design evals that surface hidden risks, and integrate gating mechanisms into CI/CD pipelines. By the end of the program, you’ll be able to scale evals across products, embed responsible AI practices into team culture, and confidently ship AI features that earn user trust.
This isn’t theory. You’ll build systematic eval suites, run failure mode analyses, and leave with the practical skills and portfolio-ready artifacts to prove you can safeguard AI products from launch to scale.
What you’ll learn
Modern Eval Frameworks
Define “good” for AI outputs with trust-driven metrics that go beyond accuracy, aligning with adoption, compliance, and growth.
Failure Mode Taxonomies
Apply structured methods to discover and prioritize hidden risks before they erode adoption or damage user trust.
Systematic Eval Suites
Translate failures into repeatable evaluators validated against gold-standard datasets for consistent, trustworthy outputs.
Eval Gating Systems
Turn evaluators into scalable gates within deployment pipelines, with clear escalation paths when thresholds are breached.
Cross-Product Eval Playbooks
Design scalable practices to monitor drift, bias, and fairness continuously across multiple product lines.
Responsible AI Culture Plan
Create a portfolio-ready playbook to embed responsible AI practices into your org and align stakeholders around long-term trust.
Who this course is for
Product Managers
who want to operationalize trust in AI products and lead with eval-driven execution.
Senior PMs/Leads
embedding safety, fairness, and reliability into team workflows at scale.
AI Advocates
who need systematic eval suites, deployment pipeline gating, and cross-product playbooks to safeguard launches.
Prerequisites
This course is not for everyone
PMs need to have a solid understanding of AI in a product context
This is not an entry level course
Completion of AI Product Management Certification is strongly advised. Prior experience with Vibe Coding is recommended for success.
It is FOR YOU if you ...
want to master Evals, scaling from prototypes to trustworthy, production-ready AI systems.
Not sure if this certification is right for you?
Schedule a callWhat's included
Live sessions
Live build labs with your AI Product Leader.
Course materials
Full access to materials and recordings, anytime.
AI tools access
$50k+ worth of AI tools, included with Unlimited Membership.
Instructor feedback
Direct, live feedback from your Silicon Valley instructor.
Workflows & templates
The most popular AI tool workflows, templates, and prompt packs.
Certification
Industry-recognized credential on completion.
Reimbursement kit
Support and resources to get your employer to cover the cost.
Alumni community
Exclusive access to a network of 10k+ alumni.
Unlimited membership
$416
$466/mo, billed annually
- Access to all 11 live instructor-led courses
- Weekly live AI workshops
- Free access to $50k+ worth of AI tools
Buy individual course
$2,999
AI Evals Certification Curriculum
01Build trusted AI products on modern eval metrics1 live session2 labs1 artifactDay 1 Hook: "What does 'good' mean — the answer, or the path to it?"Show Details
Pre-read
Output vs. trajectory eval split
AI Evaluation Strategy Canvas
First Eval Lab Walkthrough
Eval strategy canvas — "good" defined, output vs. trajectory split
Session recording
02Master failure-mode discovery1 live session2 labs1 artifactDay 2 Hook: "What can go wrong, for chatbots and for agents?"Show Details
Pre-read
Build a Failure Taxonomy, prioritize by impact
Eval Audit Walkthrough
Failure Taxonomy Canvas
Failure Taxonomy — agent + output failure modes, prioritized
Session recording
03Translate failures into calibrated eval suites1 live session3 labs1 artifactDay 3 Hook: "How do I measure quality, and trust the judge?"Show Details
Pre-read
5-part Eval Spec, judge calibration (κ ≥ 0.6)
Trajectory Eval Lab
Judge Calibration Tool
Eval Spec Builder
Calibrated eval suite — gold set + 5-part Eval Spec
Session recording
04Ship safely with a CI eval gate1 live session3 labs1 artifactDay 4 Hook: "How do I block bad releases automatically, for real?"Show Details
Pre-read
GitHub Actions eval gate, regression golden set
CI Gate Demo
Eval Gate Mapping Tool
Launch Strategy Builder
CI eval gate — thresholds and blocking dimensions defined
Session recording
05Scale evals, online & closed-loop1 live session2 labs1 artifactDay 5 Hook: "How do evals scale, and how does production feed back in?"Show Details
Pre-read
Production traces → Error Feed → offline set → gate
Coverage Matrix Tool
Budget Crisis Tool
Closed-loop coverage & scale plan
Session recording
06Champion a culture of responsible AI1 live session2 labs1 artifactDay 6 Hook: "How do I make the closed loop how my org ships, and prove compliance?"Show Details
Pre-read
Eval ROI, pyramid memo, AI-native capstone
Ship/Hold Memo Builder
Final Pitch Prompt Generator
Ship/Hold memo — data-backed recommendation on Ascend IQ
Session recording
07Final project: ship your Ascend IQ recommendation6 artifacts1 repoCertificationShow Details
Your forked repo: all six artifacts assembled into one submission
AI Evals certification review, graded by your instructor
Certificate issued on pass
Upcoming start dates
Choose the time that best fits your schedule
November 2026
1 cohort
AI Evals
Nov 7 – Nov 21
Saturdays
7:30 AM – 12:00 PM
(America / Los Angeles)
December 2026
1 cohort
AI Evals
Dec 1 – Dec 17
Mondays and Wednesdays
6:00 PM – 8:00 PM
(America / Los Angeles)
Nothing that works? Join the waitlist or request a private cohort
Explore our flexible plans
Single Certification
Ideal for: Focused learners who want to master a specific skill
$2,999
- 1 live, instructor-led certification course
Pro Membership
Ideal for: Product managers ready to build broad expertise across multiple disciplines
$3,999
/per year
- 2 live, instructor-led certification courses
- Weekly live workshops
- Access to on-demand content
- Access to our alumni community
Unlimited Membership
Ideal for: Product professionals seeking full career transformation
$4,999
/per year
- 11 live, instructor-led certification courses
- Weekly live Workshops
- 1:1 coachingComing soon
- Free access to $50k+ worth of AI tools
- Ticket to one ProductCon conference
- Access to on-demand content
- Access to our alumni community
- Access to all newly released courses
Flexible payment options
We have flexible options to get you on your way to your Product Management goals:
Upfront payment discount
Pay for your education upfront and get a discount.
Study now, pay later
Enroll today and spread your payments over time with region-based installment options. Plan availability and terms vary depending on your country.
Employer sponsorship
Use your company’s budget for professional development or continuing education to fund your education with us.
Group enrollment discount
Enjoy 10% discount on single certifications courses for groups of 3 to 10 people from the same company. Discount available only for unlimited membership
Hear it from our alumni
"The biggest shift was getting comfortable turning rough ideas into working prototypes — actually shipping things, not just planning them."
Vikrant Shukla
Sr Product Manager at Salesforce
Thank you Product School for helping bring genAI concepts down to earth. We covered technical basics such as RAG and embedding and tokenization and Product-focused ideals such as defensive and offensive AI strategies.
Deepak Ravi
Product Manager, AI & Data at New York Life
"Through the certification, I learned a lot about AI, myself, and how to keep current with the rapid changes in the industry. No doubt that I will be a better product leader, and I look forward to building our AI PM community."
Josh Gray
Global Monetization Product at Tiktok
"This program strengthened my understanding of how successful AI products are built — not just with strong models, but with the right balance of workflow integration, trust, evaluation, and operational impact."
Debolina Das
Sr Product Manager at Amazon
"It was the right time to pick up this course when AI tools like Claude Code are changing how PMs work — and this program gave me a practical, hands-on foundation from vision to launch."
Clinton Gonsalves
Product Manager at JPMC
"Product School helped me develop critical product skills to master my job and get a product promotion. The case studies and real examples shared by my instructor helped me put into practice the frameworks that I learned and drive impact in my work."
Paulina Mwangi
Sr Director of Product at Salesforce
"Claude Code gave me a better overview of the entire process — system, roadmap, strategy — and freed up mental space to focus on the quality of the work."
Jenny Kan
Product Strategy Lead at ServiceNow
"Product School helped me learn the skills that I needed to break into product, and now I work on Product Strategy for Google Play."
Shilpi Verma
Product Lead at Google
"This certification covers everything you need to know about Product Management to jumpstart and land your first PM job. It isn’t all theory, you get to apply everything you learn to your own project and the instructors provide feedback to help you improve."
Medha Ghatikesh
Product Manager at Meta
Train the whole product org
Private cohorts on your schedule, group pricing from three seats, and everything you need to get it expensed.
Group discounts starting at 3 seats
Private cohorts, your dates and your case studies
Reimbursement letter and invoicing handled
Team progress reporting for managers
Frequently asked questions
What are AI evals?
AI evals are structured tests that check whether an AI system's output is good enough for real users: correct, safe, consistent and aligned with what the product promises. Because model output varies from run to run, teams test against sets of examples with known good answers. They score results with code checks, human reviewers or another model acting as judge, then track scores over time so regressions are caught before release.
What is the AI Evals Certification?
The AI Evals Certification is a live online course from Product School for product managers shipping production AI. It teaches you to own the definition of "good" for AI products: defining good for both the output and the steps an agent takes, finding failure modes, building calibrated eval suites, gating releases with a CI eval gate, and closing the loop on production traces. It includes 12 live hours with hands-on labs. You receive a Product School certificate at the end of the course.
What is LLM evaluation, and how is it different from traditional software testing?
Traditional tests expect the same input to produce the same output every time. LLM evaluation accepts that output varies, so it measures quality across many examples instead, and for agents it scores the whole trajectory of steps, not only the final answer. The AI Evals Certification covers the full spectrum of code checks, LLM judges and human review, and how to calibrate a judge against human labels so you can trust its scores.
Why should product managers own AI evals?
Deciding what counts as a good answer is a product decision. Accuracy, tone, safety thresholds and when to block a release all depend on the users and the business, not only on the model. In the AI Evals Certification, PMs own the gate policy: which regressions stop a release and who signs off.
Who is the AI Evals Certification for?
Product managers shipping production AI features who need a defensible answer to "how do we know it works?" It is not an entry-level course. Product School strongly advises completing the AI Product Management Certification first and recommends prior prototyping experience from the Vibe Coding Certification.
How long is the AI Evals Certification?
12 live hours, taught live online, with weekday and weekend cohorts. Every class is recorded.
What will I be able to do after the AI Evals Certification?
You will be able to:
define good for AI output and for agent trajectories
discover failure modes and turn them into a taxonomy
translate failures into calibrated eval suites, with judges checked against human labels
ship safely behind a CI eval gate that fails a change when quality regresses
run evals online on production traces in a closed loop
champion a culture of responsible AI, with governance and compliance built in
How much does the AI Evals Certification cost?
$2,999 for the AI Evals Certification on its own. Pro includes 2 live certifications for $3,999 a year, and Unlimited, at $4,999 a year, gives access to all 11 live instructor-led courses.
Should I buy AI Evals as a single course?
Only if you already have the AI foundations. The prerequisites point to AI Product Management first. Those two courses together cost $3,999 with Pro, versus $5,998 as singles. Add the recommended Vibe Coding course and you are at three courses, which makes Unlimited ($4,999, access to all 11 live instructor-led courses) cheaper than buying them separately ($8,997).
Explore other certifications
Explore other certifications: AI Product Management Certification, Agentic Workflows & Loops Certification, and Product Analytics & Experimentation Certification.