Introduction

Once contracts, decoding, style, context, tools, and validators are in place, you still need one thing to ship safely: evidence that a change will not break production. Evaluation and operations provide that evidence. This article describes a practical framework—golden tests, canary releases, and one-click rollback—that turns prompt-engineered routes into systems you can change every day without fear. We’ll define the artifacts, the gating metrics, and the minimal automation you need to graduate from “it looks good locally” to “it’s safe at scale.”

The Problem Evaluation Must Solve

Generative systems fail in ways that unit tests don’t catch:

Traditional QA by eyeball can’t keep up. You need stable, automated signals that fire before customers notice.

Core Concepts

Golden Tests (Goldens). Fixed, anonymized inputs with expected properties (not verbatim text). They encode the non-negotiables: schema, refusal/ask rules, citation coverage, banned lexicon, jurisdictional limits, and tool safety.

Canary Releases. Expose a change to a small, representative slice of traffic (typically 5–10%), measure predefined gates, and pause automatically on regression.

Rollback. Promote changes behind feature flags and keep the last “green” artifact bundle (contract, policy, decoder, validators). If gates fail, revert in one action; no rebuilds or manual edits.

Designing Golden Tests

What a Golden Encodes

Size and Coverage

Start with 30–50 cases per route and grow to 100–200 for high-risk flows or varied locales. Keep a challenge subset (10–20%) for adversarial prompts, conflicting claims, and missing fields. Rotate a few cases each sprint to avoid overfitting.

File Layout (example)

/routes/<route>/goldens/
  001_missing_field.json
  014_conflicting_claims.json
  037_tool_proposal_limits.json

Each test contains inputs, optional claim pack IDs, and property assertions. The runner returns pass/fail plus a failure taxonomy (SCHEMA, CITATION, SAFETY, TONE, LENGTH, EVIDENCE, IMPLIED_WRITE).

Building a Minimal Test Runner

Your runner should accept an artifact bundle and an input bundle, then emit structured results:

Make the runner usable locally and in CI. In CI, block merges when:

Canary Strategy That Works in Practice

Traffic Split

Start at 5–10%, stratified by locale, persona, and channel so you don’t hide regressions in a skewed cohort. Keep control and treatment simultaneous to reduce confounding.

Auto-Halt Gates (copyable)

When any gate trips, halt exposure and trigger an alert with the artifact diff and failing traces.

Duration and Power

Run until you reach N accepted outputs per key segment (often 300–1000, depending on variance). Prefer sequential tests or Bayesian monitoring for early stops; pre-register the decision rule to avoid p-hacking.

Rollback as a First-Class Feature

Treat deployment and exposure separately:

Store bundles and flags as config, not code, so reversions don’t require rebuilds. Attach a human-readable changelog to each bundle for incident triage.

Metrics & Dashboards

Track by route, locale, model, and release:

Wire alerts on CPR −2 pts, p95 +20%, $/accepted +25%, any safety incident.

Operational Playbooks

Pre-Release

During Canary

Post-Release

Worked Example (Composite)

You tighten lexicon policy to reduce hype and adjust decoding for “Proof” sections.

Goldens (CI):

Canary (10%):

Rollback Drill:

Common Pitfalls—and How to Avoid Them

Conclusion

Evaluation and ops turn prompt engineering into a repeatable practice. With golden tests that encode your guarantees, canary releases that gate exposure in the wild, and one-click rollback when reality disagrees, you can change prompts, policies, decoders, and validators quickly—without betting the business. In Part 8, we’ll close the series with Cost & Speed—how to engineer for $/accepted output and predictable latency without sacrificing quality.