AI Evals for Product Managers: How to Measure AI Quality

Why AI needs evals at all

Traditional software has tests. Given an input, you assert the exact output. Pass or fail.

AI breaks that. The same prompt can give a different answer twice, so there’s no single correct output to assert against. The old test model doesn’t work, and “it looked fine when I tried it” is not a quality bar you can ship on.

Here’s the line worth keeping close: an AI feature usually fails not because the model is weak, but because no one has evaluated it. Ship AI with no evals, and the product is quietly wrong for some users, and you won’t know until they tell you, or leave.

What an eval actually is

An eval is a structured way to measure quality when the output varies. Strip away the jargon, and it’s three things.

1. A set of test cases. Real inputs your feature will face, with a sense of what a good response looks like for each.
2. A way to score each output. A rule, a rubric, or a judge that says how good the answer was.
3. A bar to clear. The score the feature has to hit before you trust it.

Run your feature against the test cases, score the outputs, and you’ve gone from “seems fine” to “here’s how good it is, on 200 real cases.” That’s the whole idea. 

The interesting question is step 2: how do you actually score an output when there’s no single right answer?

The three ways to score an output

This is the “how to measure” part, and there are three methods. Most real eval setups use a mix, matched to what you’re checking.

1. Deterministic checks (for anything with a right answer). Plain rules, no AI needed. Is the output valid JSON? Does it contain the order number? Is it under 280 characters? Does it avoid a banned word? These are cheap, instant, and perfectly reliable, so use them wherever the thing you care about is actually checkable. Not everything is.
2. LLM-as-a-judge (for subjective quality, at scale). When “good” is a judgment call (was it on-tone, was it helpful, did it stay on-policy), you can use one model to score another model’s output against a rubric you write. You hand the judge the input, the answer, and your criteria, and it returns a score with a reason. It’s how you evaluate thousands of outputs without reading each one by hand. Two cautions: the judge can be wrong, so you validate it against human-scored examples first, and you keep its rubric as tight as the one you’d give a person.
3. Human review (the gold standard that doesn’t scale). A person reads the output and judges it. It’s the most trustworthy and the least scalable, so you spend it where it counts: scoring a sample, handling the high-stakes cases, and checking that your LLM-judge actually agrees with human taste. Human review is what you calibrate the other two against.

The skill isn’t picking one. It’s knowing which method fits which criterion: deterministic where you can, LLM-judge for the subjective bulk, humans on the sample and the risky edges.

Offline vs online evals

Two moments matter, and they answer different questions

• Offline evals happen before you ship. You run the feature against your test set in a controlled way to decide if it’s good enough to release. This is your gate.
• Online evals happen after you ship. You watch real usage, sample real outputs, and measure quality in the wild, because real users do things your test set never imagined. This is where your product analytics and a clear read on metrics vs analytics earn their keep.

You need both. Offline tells you if it’s ready. Online tells you if it’s actually working once real people get their hands on it.

Real Job of an AI Product Manager: defining “good”

Here’s where evals become a product role, not an engineering one. The hardest part is not the tooling. It’s defining “good.”

What does a good answer look like for your feature? Accurate? Safe? On-brand? Concise? Honest even when it says “I don’t know”? Those are product decisions, and no model can make them for you. Deciding what “good” means is the same muscle as any other prioritization call, the kind the core product management frameworks are built for. You set the rubric. You decide the bar. You decide what a failure even is.

In our experience, the PMs who ship trustworthy AI treat the eval rubric as a core product artifact: written early, argued over, and revised, not a checkbox at the end. The ones who skip it ship something that demos well and falls apart in production. Evals are where AI product sense becomes concrete.

A sample rubric you could write today

To make “define good” concrete, here’s a rubric a PM might write for a support-reply feature. Each line is pass or fail, scored on every output.

• Accuracy: Does the reply match our actual, current policy? (No inventing a refund window.)
• Tone: warm and concise, not robotic or defensive?
• Honesty: When unsure, does it say “let me check” rather than guess?
• No false promises: does it avoid committing to a refund or action that needs human approval?
• Completeness: Does it actually answer the question the customer asked?

That’s a real eval. You’d run it against 20 real tickets, score each reply on these five lines (deterministic where you can, an LLM-judge for tone and honesty, your own eyes on a few), and now you have a number: “on 20 cases, it passed accuracy 18 times and honesty 12 times.” Suddenly, you know honesty is your weak spot, and you have something concrete to fix. That’s the difference between “it seemed fine” and knowing.

Writing this rubric took five minutes and no code. That’s the point. The hard part was never the tooling. It was deciding what good means, and that’s your job.

Where evals fit

Evals are the quality layer that sits under everything else: prompting, retrieval, and agents. None of those are trustworthy without a way to measure them, which is exactly why AI agents need evals most: an agent that acts on a wrong answer can do real damage. A good way to build the instinct: run a product teardown on an AI feature you use and write the rubric you think its team should be scoring against.

You Need a Rubric

You don’t need eval tooling to start thinking like this. You need a rubric.

Pick any AI feature, yours or one you use, and write down what a good output looks like in five bullet points. Then find three real cases where it fails your own rubric. That exercise, defining good and hunting for failures, is exactly what evals formalize, and it’s the skill that makes a hiring manager trust you with an AI product.

Wish to kickstart your AI PM journey?  Take a look at our AI PM curriculum and free AI PM resources. But the rubric is yours to write today.

Go define “good.” The rest is just scoring.

Frequently asked questions

1. What are evals in AI?

Evals are structured tests that measure the quality of an AI feature when its outputs vary. You define good answers, build a set of real test cases, and score the outputs against a bar, using deterministic rules, an LLM as a judge, or human review, depending on what you’re checking.

2. Why can’t you test AI features like normal software?

Because AI is probabilistic. The same input can give different outputs, so there’s no single correct answer to assert against. Evals replace pass-or-fail unit tests with quality scoring across many real cases.

3. What is LLM-as-a-judge?

It’s using one model to score another model’s output against a rubric you define. It lets you evaluate quality at scale instead of hand-checking every response. You still have to validate the judge against human-scored examples, because the judge can be wrong too.

4. What is the difference between offline and online evals?

Offline evals run before launch against a test set to decide if the feature is good enough to ship. Online evals run after launch on real usage to check quality in the wild. You need both.

5. What does a PM actually do in the eval process?

The PM defines what “good” means, sets the rubric and the bar, and decides what counts as a failure. That judgment is the part only the product side can do. The scoring pipeline and tooling are engineering’s job.

6. Do you need code or special tools to start with evals?

No. You can write a five-point rubric and score 20 real cases by hand in an afternoon. Tooling helps you scale later, but the valuable part, deciding what good means and finding real failures, needs nothing but judgment.