AI-Based Data Classification Pipeline (PII Detection, Anonymization Rules)

Recommendation: Use ONNX for inference performance and to avoid hybrid language complexity. For development you can prototype with Python + HuggingFace.

Example: Detection Worker (C# pseudocode)

public class DetectionWorker : BackgroundService
{
    private readonly IDataSource _source;
    private readonly IDetector _detector;
    private readonly IDbContext _db;
    private readonly IEventProducer _producer;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var batch in _source.ReadBatchesAsync(ct))
        {
            foreach (var row in batch)
            {
                var normalized = Preprocess(row);
                var ruleMatches = _detector.RunRules(normalized);
                var mlEntities = await _detector.RunModelAsync(normalized);
                var fused = Fuse(ruleMatches, mlEntities);
                foreach (var entity in fused)
                {
                    var record = MapToDetectionRecord(entity, row);
                    await _db.DetectionRecords.AddAsync(record);
                    if (ShouldAutoAnonymize(entity))
                    {
                        var anonymized = await ApplyAnonymizerAsync(entity);
                        await SaveResult(row, anonymized);
                    }
                    else
                    {
                        await _producer.ProduceAsync("review-queue", CreateReviewEvent(record));
                    }
                }
            }
            await _db.SaveChangesAsync(ct);
        }
    }
}

Anonymization Example (Pseudonymization)

public async Task<Guid> PseudonymizeAsync(string value)
{
    var salt = GetTenantSalt();
    var key = ComputeHmac(salt, value); // stable hash per tenant
    var existing = await _db.PseudonymMap.FirstOrDefaultAsync(p => p.HashKey == key);
    if (existing != null) return existing.PseudoId;
    var encrypted = EncryptWithVault(value);
    var map = new PseudonymMap { HashKey = key, EncryptedValue = encrypted };
    _db.PseudonymMap.Add(map);
    await _db.SaveChangesAsync();
    return map.PseudoId;
}

Security note: Use KMS-managed encryption keys, audit every decrypt operation.

Implementation: ML Models

Model Choices

• spaCy (fast, pipeline-friendly) — good starting point for NER.

• Transformers (DistilBERT / RoBERTa) — higher accuracy for ambiguous entities.

• Custom CRF over token features — good for lower-resource or domain-specific tuning.

Training Data

• Public NER corpora (CoNLL) for structure; synthetic generation for domain-specific PII (generate fake names/numbers in realistic contexts).

• Labelled datasets from internal manual review (human-in-the-loop) will drastically improve performance.

Model Serving

• Convert trained model to ONNX for fast cross-platform inference.

• Host inference as a microservice with batching and GPU support.

• Provide metrics: latency, throughput, model confidence distribution.

Example Fusion Heuristic

• If regex confidence >= 0.95 → mark high confidence.

• Else if ML confidence >= 0.9 → high confidence.

• If both exist → boost to 0.99.

• If ML conf between 0.6–0.9 → review.

Tune thresholds with ROC/AUC analysis against labeled validation set.

Angular UI: Review And Rule Management

UI Responsibilities

• Show pending detection items with context (original text + highlighted tokens).

• Offer suggested action (mask, pseudonymize, ignore) with one-click.

• Allow editing of entity span or type.

• Provide rule management UI: add regex, add dictionary entries, set policy per tenant.

• Approval workflow: assign to reviewer, status updates, audit trail.

Component Outline (Angular)

• ReviewListComponent — paged list of pending items.

• ReviewDetailComponent — text viewer with highlighted spans and action buttons.

• RuleManagerComponent — create/edit/delete rules; test regex against sample text.

• PolicyEditorComponent — per-tenant anonymization rules (which PII types auto-mask, pseudonymize, etc.)

Sample HTTP Interceptor (auto attach tokens)

@Injectable()
export class AuthInterceptor implements HttpInterceptor {
  intercept(req, next) {
    const token = this.auth.getToken();
    const cloned = req.clone({ setHeaders: { Authorization: `Bearer ${token}` }});
    return next.handle(cloned);
  }
}

UX Tips

• Show confidence visually (bar or color).

• Provide quick "Accept All Low Risk" for bulk handling with audit logs.

• Support keyboard shortcuts for fast triage.

Policies And Governance

Create policy documents that define actions mapped to detection types and risk levels, for example:

Policies should be configurable per tenant and stored as versioned JSON policies.

Testing Strategy

• Unit Tests: tokenizer, regex rules, HMAC hashing, DB mapping.

• Model Validation: holdout dataset with metrics (precision, recall, F1) per PII type.

• Integration Tests: end-to-end ingestion → detection → action → audit.

• Human Review Trials: measure disagreement rates and tune thresholds.

• Load Tests: ensure model server handles expected QPS with acceptable latency.

Also run false-positive and false-negative audits periodically.

Performance And Scaling

• Rule pass is fast; run it inline.

• ML inference is heavier — batch multiple samples and use GPU if available.

• Separate pipeline: preprocess + rule pass synchronous; collect uncertain items and run batched ML inference.

• Use horizontal scaling for workers and autoscale model serving by CPU/GPU metrics.

Throughput design targets

• Latency sensitive flows (ingest → mask) need sub-second rule-based masking.

• Bulk historical scans can take hours — prioritize by risk and use distributed workers.

Privacy, Security And Legal Considerations

• Minimize exposure of raw PII inside application logs — censor values in logs.

• Encrypt PII at rest and in transit.

• Audit all access to PII (who viewed original, who approved changes).

• Implement role-based access control for review UI.

• Provide data subject request APIs (export, delete) and ensure pipeline respects them.

• Keep data retention policies and deletion workflows.

Monitoring And Observability

Metrics to capture:

• Number of detections per PII type.

• Auto-anonymize rate vs review rate.

• Average ML confidence per type.

• Review queue size and reviewer throughput.

• Time to action (how long items stay in review).

Logs and traces should include traceId, job id and detection id for end-to-end debugging.

Example End-to-End Scenario

Scenario: Bulk scan of customer support transcripts to remove PII before analytics.

1. ETL extracts transcripts into object store and emits a scan-request event.

2. Detection worker picks up a transcript, runs rule pass: finds a few emails and phone numbers → masks them immediately.

3. ML pass identifies suspected names with medium confidence → creates detection records and enqueues them in review-queue.

4. Data steward opens Angular review UI, approves most suggestions and corrects a wrong span. Approval triggers policy action: pseudonymize approved names.

5. Final masked transcript stored in safe store and analytics job consumes it.

Audit trail records the original snippet (encrypted), reviewer decisions, and the final action.

Roadmap And Improvements

• Add active learning: feed reviewer corrections back to retrain model periodically.

• Add language detection and per-language models for better performance.

• Expand to structured data sources (databases) with column-level policies and differential thresholds.

• Integrate BYOK for tenant-specific key control over pseudonymization.

Summary

An AI-based data classification pipeline is a practical, high-value capability for any organisation that stores or processes sensitive data. A production-ready pipeline combines deterministic rules, ML models, a robust .NET backend, and a focused Angular review UI. Key success factors are accuracy (low false positives/negatives), auditable actions, scalable model serving, and clear governance rules.

Start with a simple hybrid detector (regex + off-the-shelf NER), add human review for low-confidence items, and iterate with active learning.