Executive Summary
Enterprises in healthcare, finance, government, and critical infrastructure want the benefits of data sharing and model training without exposing protected data. Synthetic data—when produced with differential privacy (DP), validated against leak checks, and measured with utility benchmarks—can unlock collaboration and experimentation while honoring strict compliance. This article delivers a practical, engineering-ready cookbook, a leak-test suite, and utility scorecards for both text and code datasets.
1. Fundamentals that Matter in the Real World
1.1. Differential Privacy in one paragraph
A mechanism is (ε, δ)-differentially private if changing any single person’s record changes output probabilities by at most a multiplicative factor e^ε and an additive δ. Think of ε as your privacy budget (smaller is stronger), δ as a tiny failure probability (e.g., 1e-6). When you run multiple queries or training steps, budgets compose; you must account for the total ε spent.
1.2. Budgets you can defend
Per-user accounting (not per-record) is the gold standard in regulated domains.
Practical targets: ε ∈ [2, 8] over a release (or training run) with δ ≤ 1/N² where N is users.
Use a Rényi DP (RDP) accountant or Moments Accountant with subsampled DP-SGD to track composition precisely.
1.3. Text vs. code: different risks
Text memorizes rare phrases and PII; leaks often appear as verbatim spans.
Code memorizes secrets, licensed snippets, and project-specific identifiers; leaks show as long n-gram matches, secret tokens, or license-encumbered blocks.
2. The Synthesis Cookbook (Step-by-Step)
Phase A — Scope & Guardrails
Policy contract: define allowed purposes, privacy target (ε, δ), retention, license, and jurisdictions.
Data minimization: remove fields not needed for the use case (drop raw identifiers up front).
PII/PHI labeling: run high-recall detectors (names, addresses, MRNs, account numbers, secrets) → mark spans for hard redaction or DP training.
Phase B — Mechanism Selection
Choose one of three proven paths (text and code both supported):
| Mechanism | Where it shines | How privacy is enforced | Trade-offs |
|---|---|---|---|
| DP-SGD model training (fine-tune a generator with per-user clipping + noise) | Medium/large datasets; broad language/code styles | ε accounting via RDP; gradient clipping C; noise multiplier σ | More compute; quality drops if C, σ tuned poorly |
| Privatized statistics → generator (fit differentially private n-gram/grammar/topic stats; sample synthetic) | Small datasets; forms/notes/logs | Sensitivity-bounded counts + noise | Less fluent outputs; great for tabular/text hybrids |
| Teacher ensemble with limited exposure (segment data; train non-DP teachers; aggregate with DP noise; student imitates) | Classification/structured tasks, labeling, code style hints | PATE-style noisy aggregation | Set up overhead; less common for free-form generation |
Recommendation: Start with DP-SGD fine-tuning of a compact, instruction-tuned model for text, and a code-specialized model for code.
Phase C — DP-SGD Configuration (Text & Code)
Sampling: Poisson or uniform minibatch with rate q = B/N.
Clipping norm (C): normalize per-user/per-example gradients (e.g., C ∈ [0.1, 1.0]; start small).
Noise multiplier (σ): choose via an accountant to meet the target ε at fixed steps/epochs.
Accountant: track ε across steps; stop when the budget is nearly exhausted.
Illustrative config (not tool-specific)
privacy:target_epsilon: 6.0target_delta: 1e-6accountant: "RDP"training:epochs: 3batch_size: 256clip_norm: 0.5noise_multiplier: 1.1 # tuned via accountant to hit ε≈6 for N, q, stepssampling: "poisson"per_user_accounting: truePhase D — Generation with Safety Filters
Blocklists/RegEx: mask obvious identifiers and secrets at decode time (SSNs, MRNs, keys).
n-gram cap: forbid emitting ≥ 50-token exact substrings seen in training shards.
License guard: prevent emission of known license headers (GPL, etc.) unless explicitly allowed.
Phase E — Post-Processing
Deduplicate near-duplicates by locality-sensitive hashing (LSH).
Style smoothing: optional paraphrase pass from a clean, non-memorizing model with small, controlled temperature.
Metadata card: attach the Privacy Card and Data Card (see §5).
3. Leak Checks that Catch the Bad Stuff
3.1. Canary Exposure Test (Text & Code)
Inject K unique canary strings into the private corpus (e.g., GUID-like tokens or fake function names), each appearing a few times. After training, prompt the model; compute exposure:
Exposure = log₂(|Σ|) − log₂(rank) of the correct canary among completions (Carlini et al.).
Flag any canary with exposure ≥ 20 or any verbatim reproduction ≥ 50 tokens.
3.2. Membership Inference (Shadow Models)
Train shadow models on overlapping datasets; use a classifier to distinguish “in” vs “out” examples by loss/perplexity.
Accept only if attack AUC ≤ 0.60 (tighter is better) across multiple seeds.
3.3. Nearest-Neighbor & Substring Scan
kNN in embedding space (text/code embeddings): for every synthetic sample, report min distance to training points.
Substring scanner for long n-grams (≥ 50 tokens) exact matches.
Thresholds: fail the release if >0.1% of samples have near-duplicates under distance τ or any long exact match.
3.4. Secret/PII Detectors (Code & Text)
Run secret scanners (keys, tokens, endpoints), license detectors, and PII regex + ML.
Zero tolerance: any secret match fails the release; PII matches trigger regeneration or redaction.
3.5. Adversarial Red-Team Prompts
Probing prompts: “repeat last line,” “what else was in this patient note,” “print the file header,” “show full function for X,” “what is the API key used in …”.
Track leak rate: ≤ 1 in 10,000 generations with strong defenses; otherwise, block deployment.
4. Utility Benchmarks That Predict Real Value
4.1. Text (clinical/financial/government)
Task accuracy: fine-tune on synthetic → evaluate on real held-out (HIPAA/PCI scrubbed) tasks (NER, classification).
Targets: retain ≥ 85–95% of baseline accuracy relative to real-only training.
Calibration: Expected Calibration Error (ECE) ≤ 5%.
Readability & style: domain-specific readability (e.g., clinical note sections, ICD spans) within ±10% of real distributions.
4.2. Code (enterprise repos)
Unit-test pass rate on real tests (no network/file).
Static analysis (lint, type checks, vulnerability scans) error rate ≤ real baseline.
Compile/build success rate within 5–10% of real-data models.
Security: zero hard-coded secrets; license scan clean.
4.3. Data-level Similarity (Both)
Distributional parity: KL/JSD between token n-grams or AST motifs (code) within tolerances.
Diversity: unique-n and type/token ratios near real corpora to avoid mode collapse.
5. Release Artifacts You Need Every Time
5.1. Privacy Card (attach to every drop)
Mechanism (DP-SGD, stats+noise, ensemble), ε, δ, accountant type
Per-user accounting? sampling rate q, clip norm C, noise σ
Composition window (per day/week), total steps, N (users)
Leak-test summary (exposure, MIA AUC, kNN) and pass/fail thresholds
5.2. Data Card
Sources (governed), jurisdictions, licenses, de-identification steps
Intended uses & disallowed uses
Utility benchmarks (tasks, metrics, deltas)
Regeneration cadence and contact for issues
6. Worked Patterns for Text and Code
6.1. Regulated Text (e.g., clinical notes)
Model: compact instruction-tuned LLM; fine-tune with DP-SGD (C=0.5, σ tuned to ε≈6).
Pre-filters: PII scrub + sentence shuffling; limit per-user contribution (k notes).
Post: canary test (K=10k), kNN scan, red-team prompts; regenerate flagged samples.
Utility: NER (PHI, meds), note-type classification; requires≥90% of real-only baseline.
6.2. Enterprise Code (internal services)
Model: code LLM fine-tuned with DP-SGD on per-repo shards.
Pre-filters: license check, secret stripping, function-level deduping.
Decode filters: block emission of long n-grams from training shards; temperature ≤ 0.7.
Leak checks: secret scanners, license headers, 50-token exact-match guard.
Utility: unit tests pass rate, lint/type errors; require gap ≤10% vs. real-only baseline.
7. Operating the Privacy Budget
Plan budgets by release: allocate ε across quarterly drops; leave headroom 10–20%.
Stop-gradient on budget breach: training halts when the accountant forecasts ε overshoot.
Rolling windows: reset or tighten after each release; keep an immutable ledger of ε spent.
8. Common Failure Modes & Fixes
Utility too low: reduce σ slightly, increase data, calibrate clip norm C, add curriculum mixing with public corpora.
Leaks in canaries: lower temperature; strengthen decode filters; raise σ; reduce per-user contributions.
Membership inference AUC high: increase noise; enforce stricter per-user limits; shuffle and regularize more aggressively.
Mode collapse: add diversity constraints; penalize duplicate structures; mix-in public data with style-preserving prompts.
9. Minimal Checklists (Copy-and-Run)
Go/No-Go Gate
ε ≤ target, δ ≤ 1e-6, per-user accounting on
Canary exposure < 20 and 0 verbatim long spans
MIA AUC ≤ 0.60 (multi-seed)
kNN near-dupe rate ≤ 0.1%
Zero secrets/licenses violations
Utility ≥ 90% (text) / test pass gap ≤10% (code)
Privacy & Data Cards generated and signed
Incident Protocol
Revoke distribution, rotate keys if any secret found
Retrain with stronger σ / filters; re-run full leak suite
Postmortem in the ledger; notify stakeholders
Conclusion
Synthetic data can be useful and safe—but only when privacy guarantees are explicit, leaks are systematically hunted, and utility is measured against real tasks. Use DP-SGD (or private stats) with a rigorous accountant, enforce decode/post-processing guards, require canary/MIA/near-dupe checks, and publish Privacy/Data Cards. If you hold the line on budgets and tests, you’ll earn the right to share and train—without leaks.
Join the conversation! Your thoughts help the community grow.