Write Your First AI Eval From a Failure You Didn't Predict

Most people test their AI with the inputs they can imagine. Dejan lets a room bring the ones they can't.

Dejan Krstic, Group Product Manager at Spotify, builds a live AI product from a single sentence and puts it straight in front of the room. Everyone hits it at once from their phones, and the log fills with inputs nobody predicted, including Dejan. Together you open it, judge real traces as pass or fail, and find the room doesn't agree on what counts as a failure. One eval goes in from a failure that was actually there, gets pushed live, and holds. One product, one session, screen shared the entire time.

No slides. You take part from your own phone and leave with one eval you wrote yourself.

What you'll walk away with:

1. Testing your own AI doesn't scale. You type the handful of inputs you can imagine and it behaves, so you ship. Real users type the ones you never thought of, and that's exactly where it breaks.
2. One eval, written from a failure you didn't predict. You take something the room actually broke, turn it into a written test, and push it into the live product. Then you watch it catch the failure on the next run.
3. Why two sensible people disagree about the same output. Half the room says ship it, half says that should never go out. That split is the reason the criterion has to be written down, not left in your head.