The demo trap
Every AI feature demos well. The prompt was chosen by the person who built it, the example was the one that worked, and the audience has no basis for comparison. That is why so many features die between the demo and production: nobody ever defined what working meant.
An evaluation is simply that definition, written down before the build, as a set of inputs with expected properties. It converts an opinion about quality into something a change can be tested against.
Start with twenty real examples
Pull twenty genuine inputs from the workflow you’re automating — real tickets, real briefs, real records, including the ugly ones. Twenty is enough to expose the main failure modes and small enough that a human can review the outputs in an afternoon.
Curated examples are worse than useless here, because they encode the assumption you are trying to test. The messy input that nobody wants in the eval set is usually the one that decides whether the feature survives.
Define failure more carefully than success
Good answers vary; bad answers cluster. It is far more tractable to enumerate the failures you refuse to ship — invented facts, wrong customer, missing escalation, leaked internal note — than to describe an ideal response. Score against those, and treat any occurrence as blocking rather than as a percentage.
Ship criteria should read like a list of things that must never happen, not a target accuracy.
Keep a human in the scoring loop at first
Automated graders are useful once you know what you’re grading, and misleading before that. Start with a person scoring twenty outputs, learn what the disagreements are actually about, then automate the parts that turn out to be mechanical.
Re-run it on every change
Prompts, models, retrieval sources, and upstream data all drift. An evaluation that runs once is a document; one that runs on every change is a safety net. This is the difference between a feature that improves and a feature that quietly degrades until someone complains.
It is also what makes model upgrades a routine decision rather than a leap of faith — you can answer “is the new one better for us?” with evidence from your own workflow instead of a public benchmark.