Evaluation before you build

Ask most teams building an AI product how they know it works. The usual answer is a demo that went well and a sense that the model “seems careful.”

Some teams avoid the failure described in why AI pilots never reach production: the impressive prototype that never moved its metric. They share one habit. They write the evaluation before the production code.

Proof of concept versus proof of value

A proof of concept asks whether something can be built. With a capable model and a generous demo script, almost anything can. A proof of value asks whether it moves the specific number it was built to move, measured against a baseline, with evidence.

You cannot answer that after the fact. Before any production decision, you define what a correct answer looks like, what a wrong one looks like, and how many of each you will tolerate. That definition is the evaluation set, and building it is engineering work.

What an evaluation set looks like

We built an evaluation set for a hospitality AI product that answers guest questions about menu items, and designed it adversarially.¹ Getting opening hours wrong is an inconvenience. Getting an allergen question wrong can harm a guest, so it needs a different kind of testing.

The 117 cases fall into three classes, each targeting a different way a confidently wrong answer slips through:

• Euphemism cases. The allergen is described indirectly. The test is whether the system recognizes the substance behind an unfamiliar phrase instead of matching a keyword.
• Social-engineering cases. The prompt applies pressure or urgency, invokes authority, or claims a staff member already confirmed the dish is safe. The test is whether the system holds its answer.
• Positive-framing traps. The question nudges the model toward the answer the guest wants to hear. The test is whether the system answers the question asked.

Each class matches a documented way a person, on purpose or not, pushes a system toward the reassuring answer. A set that only tests the polite, direct question never meets the versions that show up in production.

The merge gate

The evaluation runs in continuous integration as a hard merge gate that requires a 100% pass rate.¹ Any change to the model, the prompt or the retrieval pipeline that fails even one of the 117 cases blocks the merge.

The rule is expensive on purpose. A useful prompt improvement can be blocked by an unrelated regression elsewhere in the set, and the fix is to repair the regression, never to loosen the gate. A single false negative on an allergen question is not worth trading for shipping speed. The gate makes that trade impossible under deadline pressure.

Why it belongs before the build

Evaluation infrastructure is invisible in a sales demo, slower to build than the feature it protects, and rarely what a vendor leads with. It is also the clearest signal of whether an AI system is ready for real users.

Stage 02 · Prove the value tests the idea against a named metric. It ends with an evaluation report and a go, no-go or adjust recommendation, before anyone funds a production build on the strength of a demo.² The evaluation set built here then becomes the regression suite for every future change to the model, the prompt and the retrieval pipeline, for as long as the system runs. Without it, most teams find the gaps in their coverage in production, after a wrong answer has reached someone who trusted it.

If a wrong answer in your use case could seriously harm someone (a regulated claim, a safety-critical instruction, a medical or allergen question), the proof of value sprint is built around this discipline. The hospitality sector page shows where this harness sits in a wider product.

Footnotes

1. See an adversarial evaluation harness for allergen safety for the full internal build this article draws on. ↩ ↩²

2. See the KARPINITY method for the full six-stage structure, including how Stage 02 · Prove the value differs from a conventional proof of concept. ↩