LLMs Feature Flags: Canary, Rollback, and Contract SemVer

Shipping AI should look and feel like shipping software. That means treating prompts, policies, and context rules as versioned artifacts; gating releases with tests; and using canaries and rollbacks when reality bites. Feature flags for LLMs let you separate deployment from exposure, so you can ship often, learn safely, and reverse instantly—without torching user trust.

Prompt/Policy Versioning

Prompts and policies aren’t strings; they’re product surfaces. Version them with semantic intent: major when behavior guarantees change, minor when defaults or copy change, patch for fixes. Keep each contract alongside a machine-readable schema and acceptance tests. Store diffs like code, not slides, and make “what changed and why” legible to engineering, product, and risk.

A practical pattern is to declare a Contract that binds model behavior and an adjacent Context Spec that binds what evidence is eligible. Together they define the runtime policy—what sources can be used, how to rank and tie-break, when to abstain, and what the output must look like.

{
  "contract_id": "refund.v2.1.0",
  "model": "gpt-5",
  "policies": {
    "freshness_days": 60,
    "rank": ["retrieval_score", "effective_date(desc)"],
    "prefer_sources": ["policy", "contract", "system"],
    "conflict": "surface_both",
    "abstention": {"min_coverage": 0.7}
  },
  "output_schema": {
    "type": "object",
    "required": ["answer", "citations", "uncertainty", "rationale"],
    "properties": {
      "answer": {"type":"string"},
      "citations":{"type":"array","items":{"type":"string"}},
      "uncertainty":{"type":"number","minimum":0,"maximum":1},
      "rationale":{"type":"string"}
    }
  }
}

Version numbers map to guarantees. Breaking changes—like switching conflict handling from “harmonize” to “surface both”—deserve a major bump, a migration note, and a canary.

Golden Traces

Golden traces are anonymized, representative user journeys that never change except for explicit refresh windows. They anchor expectations in reality and let you compare versions without guesswork. A good set spans easy wins, edge cases, and “hard failures” that must refuse or escalate.

Each trace carries its inputs (user query, context pack, tenant/region), expected behaviors (schema validity, abstention when fields are missing, no cross-tenant access), and tolerances (latency, token budget). When you trial a new contract or prompt variant, you replay the same traces and assert that key guarantees still hold.

{
  "trace_id": "T-eligibility-009",
  "tenant": "enterprise-us",
  "pack_ref": "s3://packs/policy-2025-10-02/refund-eligibility.json",
  "expectations": {
    "must_abstain_if_missing": ["fare_class"],
    "policy_version_at_least": "2025-10-02",
    "max_latency_ms": 1200,
    "max_tokens": 3000
  },
  "assertions": [
    {"type":"schema_valid"},
    {"type":"mentions_source", "id":"policy:2025-10-02#threshold"},
    {"type":"no_harmonization_on_conflict"}
  ]
}

Golden traces convert “it feels better” into “it passes the contract.” They also become forensic tools when behavior drifts.

Pack Replays in CI

If prompts and policies are code, they get CI. Pack replays take recorded context packs and run them through the proposed contract to check grounded accuracy, citation precision/recall, policy adherence, abstention quality, latency, and cost. Failures block merges exactly like unit tests.

A minimal pipeline:

name: llm-contract-ci

on: [pull_request]

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install replay runner
        run: pip install llm-replay
      - name: Run golden traces
        run: llm-replay run \
             --contract contracts/refund.v2.1.0.json \
             --packs s3://packs/refund/*.json \
             --traces traces/refund/*.json \
             --report out/report.json
      - name: Enforce gates
        run: llm-replay enforce \
             --report out/report.json \
             --min_grounded_accuracy 0.95 \
             --min_citation_precision 0.9 \
             --min_adherence 0.98 \
             --max_latency_ms_p95 1500 \
             --max_cost_per_answer 0.0025

Pack replays turn sporadic offline evals into a standing safety net. Any PR that degrades adherence or blows cost/latency budgets fails early—before users feel it.

Rollback Rules

Canaries and rollbacks operationalize humility. You expose a new contract to a small slice of traffic, watch the right leading indicators, and cut over or back with one toggle. The art is picking rollback conditions that are sensitive and ungameable: policy adherence, abstention quality, citation precision, and tail latency, not just click-through.

A pragmatic rule set:

Canary scope: 5–10% of tenants or traffic, stratified by region and workload.
Warm-up: Ignore the first N minutes to stabilize caches, then evaluate windows (e.g., rolling 15 minutes).
Hard rollback triggers:
- Adherence score drop ≥ 2 points vs. baseline.
- Citation precision < 0.85 or recall < 0.9.
- p95 latency > SLO by 20% for two consecutive windows.
- Abstention underuse (guesses where the old contract abstained) > threshold.
Soft triggers: Cost-per-answer up > 25% with no accuracy gain; revert unless justified.
One-click revert: A feature flag flips traffic back to the previous contract; stateful agents drain gracefully.
Postmortem: Attach canary dashboards, golden-trace diffs, and pack samples to a short write-up; encode the lesson as a new CI gate.

Feature flags decouple deploy from decision. If a change is wrong for now, rollback is cheap; if it’s wrong everywhere, CI will block it next time.

Bottom line: Treat prompts and policies like software. Give them versions, tests, and flags. Use golden traces to lock in reality, replay packs in CI to prevent regressions, and wire canary + rollback so learning is safe. Models will evolve, but your ability to ship reliable behavior on purpose comes from discipline in versioning, testing, and control—not from raw parameter count.