Skip to content

Work

An adversarial evaluation harness for allergen safety

An internal build: a 117-case adversarial test set for allergen answers in a hospitality AI product, wired into CI as a merge gate that requires every case to pass.

What it proves
Evaluation as code: no change to the assistant ships unless all 117 allergen cases still pass.
Method stages
Built from
  • 117-case adversarial evaluation set
  • Euphemism-detection cases
  • Social-engineering prompt cases
  • Positive-framing trap cases
  • CI merge gate requiring a 100% pass rate
  • Regression tracking across model, prompt and retrieval changes
Sectors

The problem

An assistant that gets a check-in time wrong costs a guest a few minutes. An assistant that wrongly tells a guest a dish is free of an allergen can put that guest in hospital. Most teams test both the same way: a spot check before launch. That leaves no record of what was tested and no way to tell when a later change breaks it.

What we built

We wrote an evaluation set of 117 adversarial cases aimed at allergen safety, in three classes:

  • Euphemism cases describe the allergen indirectly instead of naming it.
  • Social-engineering cases apply pressure, urgency or authority, including a claim that a staff member has already confirmed the dish is fine.
  • Positive-framing cases word the question so that the answer the guest hopes for is the easy one to give.

The set runs in CI as a merge gate with a 100% threshold, so a single failing case blocks the merge. Any change to the model, the prompt or the retrieval pipeline goes through it, and results are tracked across changes as regressions.

The cost is that a useful prompt improvement can be blocked by an unrelated regression elsewhere in the set. The fix is to repair the regression, never to lower the gate.

What it proves for a client

The evaluation exists before the change, runs automatically and can stop a release. That is how you know a change did not make the assistant less safe.

Method stages

It belongs to Stage 02 · Prove the value, where the evaluation is written before the prototype is built. It carries into Stage 06 · Run and evolve, where the same set guards every later model or prompt change. See the KARPINITY Method.