AI Evals for Product Managers: How to Measure AI Quality
Evals are how you judge whether an AI feature is doing its job when the same input can produce different […]
Why AI needs evals at all
Traditional software has tests. Given an input, you assert the exact output. Pass or fail.
AI breaks that. The same prompt can give a different answer twice. There is no single correct output to assert against. So the old test model does not work, and “it looked fine when I tried it” is not a quality bar you can ship on.
Here is the line worth keeping close: an AI feature fails not because the model is weak, but because you never evaluated it. Ship AI with no evals and the product is quietly lying to your users, and you will not know until they tell you.
What an eval actually is
- An eval is a structured way to measure quality when the output varies. Strip away the jargon, and it is three things.
- A set of test cases. Real inputs your feature will face, with a sense of what a good response looks like.
- A way to score each output. A rule, a rubric, or a judge that says how good the answer was.
- A bar to clear. The score the feature has to hit before you trust it.
Run your feature against the test cases, score the outputs, and you have gone from “seems fine” to “here is how good it is, on real cases.” That is the whole idea.
LLM-as-a-judge, explained
The obvious problem: if outputs vary and there is no single right answer, who scores them? Scoring thousands of responses by hand does not scale.
A common answer is “LLM as a judge.” You use one model to evaluate another model’s output against a standard you define. You give the judge the question, the answer, and your rubric, and it scores it: was this accurate, was it on-tone, did it follow the rules?
It is not perfect, and you have to check the judge itself. But it lets you evaluate quality at scale, which is the only way to keep up with a probabilistic product.
Offline vs online evals
Two moments matter, and they answer different questions.
- Offline evals happen before you ship. You run the feature against your test set in a controlled way to decide if it is good enough to release. This is your gate.
- Online evals happen after you ship. You watch real usage, sample real outputs, and measure quality in the wild, because real users do things your test set never imagined.
You need both. Offline tells you if it is ready. Online tells you if it is actually working once real people get their hands on it.
The PM’s real job in all of this
Here is where it becomes a product role, not an engineering one. The hardest part of evals is not the tooling. It is defining “good.”
What does a good answer look like for your feature? Accurate? Safe? On-brand? Concise? Helpful even when it says “I don’t know”? Those are product decisions, and no model can make them for you. You set the rubric. You decide the bar. You decide what a failure even is.
Across the PMs/aspiring PMs we mentor at HelloPM, the ones who ship trustworthy AI treat the eval rubric as a core product artifact, written early, argued over, and revised, not a checkbox at the end. The ones who skip it ship something that demos well and falls apart in production. Evals are where AI product sense becomes concrete.
A sample rubric you could write today
To make “define good” concrete, here is a rubric a PM might write for that support-reply feature. Each line is pass or fail, scored on every output.
- Accuracy: does the reply match our actual, current policy? (No inventing a refund window.)
- Tone: warm and concise, not robotic or defensive?
- Honesty: when unsure, does it say “let me check” rather than guess?
- No false promises: does it avoid committing to a refund or action that needs human approval?
- Completeness: does it actually answer the question the customer asked?
That is a real eval. You would run it against 20 real tickets, score each reply on these five lines, and now you have a number: “on 20 cases, it passed accuracy 18 times and honesty 12 times.” Suddenly you know honesty is your weak spot, and you have something concrete to fix. That is the difference between “it seemed fine” and knowing.
Writing this rubric took five minutes and no code. That is the point; the hard part was never the tooling. It was deciding what good means, and that is your job.
Where evals fit
Evals are the quality layer that sits under everything: prompting, RAG, agents. None of those are trustworthy without a way to measure them.
To see where evals sit in the broader skill set, read the AI PM skill stack. And because evals measure quality but do not tell you if the feature is winning, pair this with how to measure the success of an AI feature.
Frequently asked questions
- What are evals in AI?
Evals are structured tests that measure the quality of an AI feature when outputs vary. You define good answers, build a set of real test cases, and score the model’s outputs against a bar, often using another model as the judge.
- Why can’t you test AI features like normal software?
Because AI is probabilistic. The same input can give different outputs, so there is no single correct answer to assert against. Evals replace pass-or-fail tests with quality scoring across many real cases.
- What is LLM-as-a-judge?
It is using one model to score another model’s output against a rubric you define. It lets you evaluate quality at scale instead of hand-checking every response, though you still have to validate the judge itself.
- What is the difference between offline and online evals?
Offline evals run before launch against a test set to decide if the feature is good enough to ship. Online evals run after launch on real usage to check quality in the wild. You need both.
- What does a PM do in the eval process?
The PM defines what “good” means, sets the rubric and the bar, and decides what counts as a failure. That judgment is the part only the product side can do; the scoring and tooling are engineering’s job.
Where to start
You do not need eval tooling to start thinking like this. You need a rubric.
Pick any AI feature, yours or one you use, and write down what a good output looks like in five bullet points. Then find three real cases where it fails your own rubric. That exercise, defining good and hunting for failures, is exactly what evals formalise, and it is the skill that makes hiring managers trust you with an AI product.
When you want to run real evals and build trustworthy AI features with feedback, that is what the HelloPM AI Product Management cohort is for. But the rubric is yours to write today.
