The pattern is familiar: a compelling demo, an enthusiastic stakeholder, and six months later a feature nobody trusts.

Build the set before you build the feature

Write down fifty to two hundred realistic inputs with expected behaviour, including the ambiguous ones, the adversarial ones and the ones where the correct answer is "I do not know". Freeze it. Every prompt change, model upgrade or retrieval tweak gets scored against the same set, so you find out whether you improved something or simply moved the failures.

Judge the right thing

• For extraction, exact match on fields is fine and cheap.
• For summarisation, human review of a sample beats any automated score, and a model grading its own family is not an independent opinion.
• For anything user-facing, measure the rate of confident wrong answers separately. Those cost far more trust than a refusal.

And instrument production. Offline scores predict online behaviour badly, because real users ask things your set never imagined.