Introduction

Every enterprise handles sensitive data — names, emails, IDs, addresses, medical notes, financial numbers. Regulations (GDPR, CCPA, HIPAA) and simple business risk demand you find personally identifiable information (PII) quickly and treat it correctly. Manual scans are slow and error-prone. An AI-based data classification pipeline fast-tracks discovery, applies anonymization or masking rules, and provides human-review flows for edge cases.

This article walks you through designing and implementing a production pipeline that:

I assume you want a practical guide you can implement in enterprise systems: clear architecture, data models, sample code, testing strategy, and operational guidance.

Goals and Non-Goals

Goals

Non-Goals

High-Level Architecture (Block Diagram B)

             ┌────────────────────────┐
             │   Data Sources         │
             │ (DBs, Files, Streams)  │
             └──────────┬─────────────┘
                        │
                        ▼
               ┌────────────────────┐
               │ Ingest Layer       │
               │ (Batch / Stream)   │
               └──────────┬─────────┘
                          │
            ┌─────────────▼────────────┐
            │ Preprocessing & Normal.  │
            │ (tokenize, normalize)    │
            └─────────────┬────────────┘
                          │
             ┌────────────▼────────────┐
             │ Hybrid Detector         │
             │ - Regex & Heuristics    │
             │ - ML Model (NER)        │
             └────────────┬────────────┘
                          │
         ┌────────────────┴────────────────┐
         │ Confidence Scorer & Risk Engine │
         └────────────┬─────────────┬──────┘
                      │             │
                      ▼             ▼
       ┌────────────────────┐  ┌──────────────┐
       │ Auto-Anonymizer     │  │ Review Queue │
       │ (mask/replace/hash) │  │ (UI + Human) │
       └─────────┬───────────┘  └──────────────┘
                 │                        │
                 ▼                        ▼
        ┌────────────────────┐    ┌────────────────┐
        │ Output Store (Safe)│    │ Audit + Logs   │
        │ (masked / pseud.)  │    │ (who, when)    │
        └────────────────────┘    └────────────────┘

Key Concepts and Decisions

Hybrid Detection: Rules + ML

Confidence And Risk

Every detection must carry a confidence score and a risk label (Low/Medium/High). Risk drives action:

Anonymization Modes

Human-in-the-Loop

Low confidence items should be pushed into a review UI where privacy officers or data stewards can approve, adjust classification, or add suppression rules.

Auditability and Explainability

Record each detection event (original value, detected type, detector used, score, action taken, operator decisions). This is essential for compliance audits.

Data Model (Core Tables)

Simplified SQL schema for meta:

CREATE TABLE DetectionJob (
  JobId UNIQUEIDENTIFIER PRIMARY KEY,
  SourceName NVARCHAR(200),
  StartedAt DATETIME2,
  CompletedAt DATETIME2 NULL,
  Status NVARCHAR(20) -- Pending, Running, Completed, Failed
);

CREATE TABLE DetectionRecord (
  RecordId BIGINT IDENTITY PRIMARY KEY,
  JobId UNIQUEIDENTIFIER,
  SourceKey NVARCHAR(400), -- pointer to source row or file path
  FieldName NVARCHAR(200) NULL,
  OriginalSnippet NVARCHAR(MAX),
  DetectedType NVARCHAR(100), -- EMAIL, SSN, NAME, CREDIT_CARD
  Confidence FLOAT,
  RiskLevel NVARCHAR(20), -- Low, Medium, High
  ActionTaken NVARCHAR(50), -- Masked, Pseudonymized, Reviewed
  ActionedAt DATETIME2 NULL,
  ActionedBy NVARCHAR(200) NULL,
  AuditJson NVARCHAR(MAX) NULL -- detector traces, regex matched groups, model logits
);

For pseudonymization mapping:

CREATE TABLE PseudonymMap (
  PseudoId UNIQUEIDENTIFIER PRIMARY KEY DEFAULT NEWID(),
  HashKey NVARCHAR(200), -- salted hash of original
  EncryptedValue VARBINARY(MAX), -- encrypted original value
  CreatedAt DATETIME2 DEFAULT SYSUTCDATETIME()
);

Encryption and key management must use Key Vault/HSM in production.

Pipeline Flow (Detailed)

  1. Ingest

    • Batch: schedule ETL jobs to extract columns / files.

    • Stream: subscribe to CDC topics, Kafka, or messaging streams.

    • For files (PDF/DOCX), run OCR first (Tesseract or cloud OCR) to get text.

  2. Preprocess

    • Normalize whitespace, remove HTML tags, canonicalize formats (phone internationalization).

    • Tokenize into sentences/words; keep offsets for redaction.

  3. Rule-Based Pass

    • Run fast regex checks and dictionary lookups.

    • Validate numbers (credit-card Luhn, IBAN patterns, checksum validation).

    • Mark exact matches as high confidence.

  4. ML Model Pass

    • Run a Named Entity Recognition model (fine-tuned for PII).

    • Models can be Transformers (DistilBERT, RoBERTa), CRF over tokens, or lighter spaCy pipelines for speed.

    • Capture per-entity confidence logits.

  5. Confidence Fusion & Risk Scoring

    • Combine rule and ML signals with heuristics: e.g., regex match + model support → raise confidence.

    • Apply contextual checks: presence of "SSN" label near numbers increases risk.

    • Assign final action via policy rules.

  6. Auto-Anonymize or Queue for Review

    • If auto-action → apply anonymization per policy (mask, pseudonymize).

    • If review → emit event to review queue with original snippet and suggested action.

  7. Persist and Audit

    • Store detection record and mapping (if pseudonymized) securely.

    • Push audit logs to immutable store (append-only) with operator metadata.

  8. Downstream

    • Safe data flows to analytics, exports, or storage.

    • Provide APIs for data retrieval which enforce decryption/authorized token resolution.

Implementation: .NET Backend

Technology Choices

Recommendation: Use ONNX for inference performance and to avoid hybrid language complexity. For development you can prototype with Python + HuggingFace.

Example: Detection Worker (C# pseudocode)

public class DetectionWorker : BackgroundService
{
    private readonly IDataSource _source;
    private readonly IDetector _detector;
    private readonly IDbContext _db;
    private readonly IEventProducer _producer;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var batch in _source.ReadBatchesAsync(ct))
        {
            foreach (var row in batch)
            {
                var normalized = Preprocess(row);
                var ruleMatches = _detector.RunRules(normalized);
                var mlEntities = await _detector.RunModelAsync(normalized);
                var fused = Fuse(ruleMatches, mlEntities);
                foreach (var entity in fused)
                {
                    var record = MapToDetectionRecord(entity, row);
                    await _db.DetectionRecords.AddAsync(record);
                    if (ShouldAutoAnonymize(entity))
                    {
                        var anonymized = await ApplyAnonymizerAsync(entity);
                        await SaveResult(row, anonymized);
                    }
                    else
                    {
                        await _producer.ProduceAsync("review-queue", CreateReviewEvent(record));
                    }
                }
            }
            await _db.SaveChangesAsync(ct);
        }
    }
}

Anonymization Example (Pseudonymization)

public async Task<Guid> PseudonymizeAsync(string value)
{
    var salt = GetTenantSalt();
    var key = ComputeHmac(salt, value); // stable hash per tenant
    var existing = await _db.PseudonymMap.FirstOrDefaultAsync(p => p.HashKey == key);
    if (existing != null) return existing.PseudoId;
    var encrypted = EncryptWithVault(value);
    var map = new PseudonymMap { HashKey = key, EncryptedValue = encrypted };
    _db.PseudonymMap.Add(map);
    await _db.SaveChangesAsync();
    return map.PseudoId;
}

Security note: Use KMS-managed encryption keys, audit every decrypt operation.

Implementation: ML Models

Model Choices

Training Data

Model Serving

Example Fusion Heuristic

Tune thresholds with ROC/AUC analysis against labeled validation set.

Angular UI: Review And Rule Management

UI Responsibilities

Component Outline (Angular)

Sample HTTP Interceptor (auto attach tokens)

@Injectable()
export class AuthInterceptor implements HttpInterceptor {
  intercept(req, next) {
    const token = this.auth.getToken();
    const cloned = req.clone({ setHeaders: { Authorization: `Bearer ${token}` }});
    return next.handle(cloned);
  }
}

UX Tips

Policies And Governance

Create policy documents that define actions mapped to detection types and risk levels, for example:

TypeLowMediumHigh
EMAILannotatepseudonymizemask
NAMEannotatereviewmask
CREDIT_CARDmaskmaskmask

Policies should be configurable per tenant and stored as versioned JSON policies.

Testing Strategy

Also run false-positive and false-negative audits periodically.

Performance And Scaling

Throughput design targets

Privacy, Security And Legal Considerations

Monitoring And Observability

Metrics to capture:

Logs and traces should include traceId, job id and detection id for end-to-end debugging.

Example End-to-End Scenario

Scenario: Bulk scan of customer support transcripts to remove PII before analytics.

  1. ETL extracts transcripts into object store and emits a scan-request event.

  2. Detection worker picks up a transcript, runs rule pass: finds a few emails and phone numbers → masks them immediately.

  3. ML pass identifies suspected names with medium confidence → creates detection records and enqueues them in review-queue.

  4. Data steward opens Angular review UI, approves most suggestions and corrects a wrong span. Approval triggers policy action: pseudonymize approved names.

  5. Final masked transcript stored in safe store and analytics job consumes it.

Audit trail records the original snippet (encrypted), reviewer decisions, and the final action.

Roadmap And Improvements

Summary

An AI-based data classification pipeline is a practical, high-value capability for any organisation that stores or processes sensitive data. A production-ready pipeline combines deterministic rules, ML models, a robust .NET backend, and a focused Angular review UI. Key success factors are accuracy (low false positives/negatives), auditable actions, scalable model serving, and clear governance rules.

Start with a simple hybrid detector (regex + off-the-shelf NER), add human review for low-confidence items, and iterate with active learning.