DLP accuracy validation

The Arbitex DLP pipeline is only as valuable as its detection accuracy. False positives interrupt legitimate workflows, erode user trust, and generate operational noise. False negatives allow sensitive data to reach model providers undetected — the outcome the entire system exists to prevent. Measuring accuracy systematically, tracking it over time, and gating deployments on objective thresholds is what separates a DLP system from a DLP checkbox.

Tier 1 — Regex accuracy headline

Metric	Score
Precision	98.97%
Recall	99.30%
F1 Score	99.13%
Corpus	92,944 examples · 71 entity types

Measured on the easy+medium corpus using isolated per-pattern evaluation.

Tier 3 — DeBERTa v4 accuracy (production)

Metric	Score
Accuracy	99.48%
Model	DeBERTa-v3-large (304M params)
Corpus	281K examples · 35 entity types
Training data	2.8M examples

Full-pipeline NLI validation accuracy. Per-entity breakdown: DLP Accuracy — Methodology & Results.

This guide describes the accuracy validation framework for the 5-tier Arbitex DLP pipeline. The framework specification is being implemented in platform-0076. Corpus design, metric definitions, CLI tooling, CI gates, and baseline management are all covered here. Actual benchmark results will be published in this guide once platform-0076 ships.

Note

Phase 1 of this guide documented methodology. Phase 2 adds isolated vs integrated testing, shadowing analysis, confidence calibration findings, pattern fix methodology, accuracy target thresholds, and guidance for interpreting accuracy reports. Measured benchmark numbers from the golden corpus are shown as illustrative examples; actual results from your deployment will vary based on configuration and corpus composition.

---

Introduction

Why accuracy measurement matters

The DLP pipeline makes a binary decision on every token of every request and response that flows through Arbitex. It either flags content as sensitive or it does not. That decision has two failure modes:

• False positive (FP): non-sensitive content flagged as sensitive. A user’s internal project codename that matches a regex for a credential format, a test email address in a developer prompt, a sample SSN from official documentation. Each FP either redacts content the user needed or blocks a request that was harmless. At scale, even a 1% FP rate against clean traffic generates significant noise.

• False negative (FN): sensitive content that passes through undetected. A credit card number formatted with spaces instead of dashes. An AWS key embedded mid-sentence in a customer support transcript. An SSN in a foreign-locale format. Each FN is a potential data leak.

Configuration — tier enablement, custom rules, sensitivity thresholds — determines what the pipeline is capable of detecting. Accuracy measurement determines what it actually detects when faced with real-world variation. The two questions are complementary but distinct. You can have a perfectly configured pipeline that still underperforms on edge cases, and you can have a high-recall pipeline that fires too broadly. Accuracy measurement tells you where you stand on both dimensions simultaneously.

Accuracy vs. configuration

The DLP Pipeline Configuration guide covers how to enable and tune each tier — adding custom regex rules, adjusting NER thresholds, setting DeBERTa sensitivity, and configuring policy actions. This guide covers something different: given your current configuration, how well is the pipeline actually performing?

The accuracy suite runs your live configuration against a curated labeled dataset and produces objective metrics. When you change your configuration — adding a new regex rule, updating the NER model, adjusting confidence thresholds — you run the suite again and compare against a stored baseline. If metrics improve, you commit the baseline update. If they degrade, you investigate before merging.

Relationship to the 5-tier pipeline

The accuracy validation framework covers all five detection tiers:

Tier	Component	Measurement focus
Tier 1	Regex matching	Pattern coverage, Luhn/checksum correctness, FP rate on syntactically similar benign text
Tier 2	NER (spaCy)	Contextual recall on ambiguous entities, FP rate on common English words
Tier 3	DeBERTa NLI	Contextual precision on low-confidence Tier 1/2 hits, confidence calibration
Tier 4	CredInt	Hit rate on known-compromised credentials, FP rate on high-entropy non-compromised tokens

Tier 4 (Credential Intelligence) operates asynchronously and is evaluated separately from the synchronous pipeline metrics. See Credential Intelligence for architectural context.

---

Evaluation framework

Golden test corpus

The foundation of accuracy measurement is a curated labeled dataset called the golden test corpus. The corpus consists of text samples where the expected detection outcome is known in advance. Every sample is hand-labeled and reviewed before it enters the corpus.

The corpus is organized into three sample categories:

Positive samples — text that contains real sensitive data of a specific entity type and should be detected. These samples exercise recall. A positive sample that is not detected is a false negative.

"My credit card is 4532015112830366, expires 09/28."
Expected: CREDIT_CARD detected, Tier 1, confidence 1.0

Negative samples — text that looks superficially like sensitive data but is not. These samples exercise precision. A negative sample that is flagged is a false positive.

"The product ID is 4532-0151-1283-0366 (non-Luhn-valid test ID)."
Expected: no detection

Edge case samples — text that is genuinely sensitive but formatted, embedded, or presented in ways that challenge the pipeline. These include partial matches, data embedded in prose, multi-language content, obfuscated formats, and content that straddles entity-type boundaries.

"Wire transfer to account ending in ssn: 078-05-1120 (test value from IRS Publication 1346)."
Expected: SSN detected, Tier 1 or Tier 2, notes: "embedded prose, IRS test SSN"

Corpus JSON format

Each corpus sample is a JSON object with the following fields:

{
  "id": "cc-pos-001",
  "entity_type": "CREDIT_CARD",
  "text": "Please charge card 4532015112830366 for the order.",
  "expected_detected": true,
  "expected_tier": 1,
  "expected_confidence_min": 0.99,
  "locale": "en",
  "category": "positive",
  "notes": "Visa, Luhn-valid, mid-sentence"
}

Field	Type	Description
id	string	Unique identifier, format {entity}-{category}-{seq}
entity_type	string	Entity type constant (see entity types below)
text	string	The raw text to run through the pipeline
expected_detected	boolean	Whether a finding should be produced
expected_tier	integer or null	Which tier is expected to produce the finding (null = any tier acceptable)
expected_confidence_min	float or null	Minimum expected confidence score (null = not checked)
locale	string	BCP-47 locale tag for the sample
category	enum	positive, negative, or edge_case
notes	string	Human-readable description of what the sample tests

The corpus file is stored at tests/dlp/corpus/golden.jsonl — one JSON object per line. Separate per-entity-type files exist at tests/dlp/corpus/{entity_type}.jsonl for maintainability; the build step merges them into golden.jsonl automatically.

Entity types covered

The corpus includes samples for every entity type in the platform:

Entity type	Description
CREDIT_CARD	All major networks, with and without separators, Luhn-valid and Luhn-invalid decoys
SSN	US Social Security Numbers, formatted and unformatted
EIN	US Employer Identification Numbers
IBAN	International Bank Account Numbers (30+ country codes)
API_KEY	Generic API key patterns, high-entropy tokens
AWS_KEY	AWS Access Key IDs (AKIA...) and secret access keys
GCP_KEY	GCP service account JSON fragments
AZURE_SECRET	Azure SAS tokens and connection strings
BEARER_TOKEN	HTTP Authorization Bearer tokens
JWT	JSON Web Tokens
PRIVATE_KEY	PEM-encoded private key headers
EMAIL	Email addresses, including subaddress and IDN forms
PHONE	US and international phone numbers (E.164 and formatted)
NPI	US National Provider Identifier
DEA_NUMBER	DEA practitioner registration numbers
PASSPORT	Passport numbers (15+ country formats)
IP_ADDRESS	IPv4 and IPv6 addresses

Maintaining and extending the corpus

The corpus is a first-class artifact. Treat corpus modifications with the same review rigor as production code changes.

To add samples:

1. Create the sample in the appropriate per-entity file under tests/dlp/corpus/.
2. Assign an ID following the existing naming scheme.
3. Run npm run dlp:corpus:validate to check for schema errors, duplicate IDs, and unreachable expected_tier values.
4. Open a pull request. The accuracy suite runs in CI automatically; if the new sample causes a regression on existing metrics, CI fails.

To remove or modify samples:

Caution

Deleting a positive sample improves apparent recall without improving real-world recall. Never remove positive samples to fix a metric regression — fix the pipeline instead.

Corpus modifications that weaken test coverage require explicit reviewer sign-off. The PR template includes a checklist item for this.

---

Evaluation harness

The evaluation harness feeds each corpus sample through the DLP pipeline in sequence, records the result, and aggregates metrics. It runs the same pipeline code that operates in production — not a stub or mock.

Running the suite

# Full suite against all entity types
npm run dlp:accuracy

# Equivalent Python invocation (for CI environments without Node)
python -m arbitex.dlp.accuracy

# Single entity type
npm run dlp:accuracy -- --entity-type CREDIT_CARD

# Single tier
npm run dlp:accuracy -- --tier 1

# Single tier, single entity type
npm run dlp:accuracy -- --tier 2 --entity-type SSN

# Verbose: show per-sample results
npm run dlp:accuracy -- --verbose

# Output to file (default: stdout + tests/dlp/results/latest.json)
npm run dlp:accuracy -- --output tests/dlp/results/my-run.json

Environment variables

Variable	Default	Description
DLP_ACCURACY_CORPUS	tests/dlp/corpus/golden.jsonl	Path to corpus file
DLP_ACCURACY_BASELINE	tests/dlp/baselines/current.json	Baseline file to compare against
DLP_ACCURACY_THRESHOLD_FILE	tests/dlp/accuracy.config.json	Threshold configuration
DLP_ACCURACY_TIMEOUT	30	Per-sample timeout in seconds
CREDINT_SERVICE_URL	http://credint:8202	CredInt service for Tier 4 evaluation
DLP_ACCURACY_SKIP_TIER4	false	Set to true to skip CredInt evaluation (useful when CredInt is not available in test environments)

---

Metrics

Per-entity-type metrics

For each entity type, the harness computes precision, recall, and F1 using the standard information retrieval definitions.

True positive (TP): the pipeline produced a finding, and expected_detected is true.

False positive (FP): the pipeline produced a finding, and expected_detected is false.

False negative (FN): the pipeline produced no finding, and expected_detected is true.

True negative (TN): the pipeline produced no finding, and expected_detected is false.

Precision measures how trustworthy a detection is. A pipeline with high precision rarely fires on benign content.

Precision = TP / (TP + FP)

Recall measures how much sensitive content is caught. A pipeline with high recall misses little sensitive content.

Recall = TP / (TP + FN)

F1 is the harmonic mean of precision and recall. It penalizes configurations that optimize one at the expense of the other.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

False positive rate (FPR) measures the fraction of negative samples that triggered a finding.

FPR = FP / (FP + TN)

False negative rate (FNR) measures the fraction of positive samples that were missed.

FNR = FN / (FN + TP)

Example metric table (illustrative — results pending platform-0076)

The following table shows the format of the per-entity accuracy report. Values shown are illustrative placeholders; real measured values will be published here after platform-0076 is complete.

Entity type	TP	FP	FN	TN	Precision	Recall	F1	FPR
CREDIT_CARD	—	—	—	—	pending	pending	pending	pending
SSN	—	—	—	—	pending	pending	pending	pending
AWS_KEY	—	—	—	—	pending	pending	pending	pending
API_KEY	—	—	—	—	pending	pending	pending	pending
EMAIL	—	—	—	—	pending	pending	pending	pending
PHONE	—	—	—	—	pending	pending	pending	pending
IBAN	—	—	—	—	pending	pending	pending	pending
BEARER_TOKEN	—	—	—	—	pending	pending	pending	pending
JWT	—	—	—	—	pending	pending	pending	pending
PRIVATE_KEY	—	—	—	—	pending	pending	pending	pending

---

Per-tier breakdown

The harness also reports metrics broken down by which tier produced the finding. This helps you understand each tier’s contribution and identify where configuration changes have the most leverage.

Tier 1: Regex matching

Regex patterns are deterministic. A pattern either matches or it does not. This means:

• Precision is entirely determined by pattern correctness and checksum validation. If a pattern fires on benign text, the pattern is too broad or checksum validation is not implemented.
• Recall is bounded by pattern coverage. Any format variant not covered by a pattern is a miss.
• Confidence is always 1.0 for Tier 1 detections — the match is binary.

Tier 1 FP rate is the most operationally sensitive metric. Tier 1 runs on all traffic synchronously, and a high FP rate means users see frequent incorrect redactions or blocks.

Tier 2: NER (spaCy)

The spaCy en_core_web_sm model identifies named entities using contextual features. For DLP purposes, entity types like PERSON, ORG, GPE, and CARDINAL are mapped to Arbitex entity types where applicable.

• Precision is moderate for ambiguous entities. “John Smith” is a person name in most contexts, but in some contexts it is a test account or a generic placeholder. NER cannot always distinguish.
• Recall improves on entities that are too context-dependent for regex — NER recognizes “my social security number is [number]” from the surrounding text even if the number format does not match the primary pattern.
• Confidence varies. The NER model does not produce calibrated probabilities; confidence scores from Tier 2 are derived from entity type certainty and should be treated as ordinal rather than probabilistic.

Tier 3: DeBERTa NLI

The DeBERTa model runs natural language inference against a set of sensitivity hypotheses. It receives the text that contains a candidate finding from Tier 1 or 2 and produces a probability that the text contains sensitive content in context.

• Precision is the highest of any tier. DeBERTa can distinguish “the test SSN 078-05-1120 from the IRS publication” from a real SSN in a production transcript.
• Recall is constrained by the fact that Tier 3 only runs on candidates surfaced by Tiers 1 and 2. If Tier 1 and Tier 2 miss an entity entirely, Tier 3 never sees it.
• Confidence is well-calibrated — the DeBERTa model produces probabilities that closely match actual precision in holdout evaluation.
• Latency is the highest of the synchronous tiers (~150 ms per request). Tier 3 is only invoked when Tiers 1 and 2 produce findings without sufficient confidence — it does not run on clean traffic.

Tier 4: Credential Intelligence

CredInt operates asynchronously and is evaluated differently from the synchronous tiers.

• Precision depends on corpus freshness and lookup protocol. The current corpus contains an industry-leading number of compromised credential hashes. A hit means the credential has appeared in a known breach. The precision metric for CredInt answers: “of all CredInt flags, what fraction involved credentials that are genuinely compromised?”
• Recall is bounded by corpus coverage. Novel credentials that have not appeared in any breach dataset will not be flagged. This is expected behavior, not a defect.
• Latency is measured separately from pipeline latency because CredInt results arrive asynchronously. See Credential Intelligence for the full architecture.

Note

Evaluating Tier 4 requires a running CredInt microservice. In CI environments where credint:8202 is not available, set DLP_ACCURACY_SKIP_TIER4=true to skip Tier 4 evaluation. The accuracy report will note that Tier 4 results are absent.

---

Aggregate metrics

In addition to per-entity and per-tier metrics, the harness produces aggregate figures for the full corpus run:

Metric	Definition
Overall precision	TP / (TP + FP) across all entity types
Overall recall	TP / (TP + FN) across all entity types
Overall F1	Harmonic mean of overall precision and recall
Overall FPR	FP / (FP + TN) across all entity types
Overall FNR	FN / (FN + TP) across all entity types
P50 detection latency	Median wall-clock time per sample through the pipeline
P95 detection latency	95th-percentile wall-clock time per sample
P99 detection latency	99th-percentile wall-clock time per sample
Tier 3 invocation rate	Fraction of samples that reached Tier 3
Tier 4 hit rate	Fraction of CredInt-checked samples that produced a hit

---

Target thresholds

The following thresholds define what the pipeline must achieve to pass CI regression gates. Thresholds are set conservatively relative to expected performance — they define a floor, not a target.

Global thresholds

Metric	Threshold	Rationale
Precision	≥ 0.95	At this rate, 1 in 20 detections is a false positive — operationally acceptable
Recall	≥ 0.90	At this rate, 1 in 10 sensitive samples is missed — improvement is expected but this is the gate
F1	≥ 0.92	Derived from the precision/recall thresholds
FPR	≤ 0.02	At most 2% of clean samples trigger a detection

Per-entity thresholds

Some entity types have different threshold requirements based on their risk profile and the difficulty of the detection problem:

Entity type	Precision	Recall	F1	FPR	Notes
CREDIT_CARD	≥ 0.98	≥ 0.95	≥ 0.96	≤ 0.01	Luhn validation makes high precision achievable
SSN	≥ 0.95	≥ 0.92	≥ 0.93	≤ 0.02	Nine-digit patterns are moderately ambiguous
AWS_KEY	≥ 0.99	≥ 0.98	≥ 0.98	≤ 0.005	AKIA prefix is highly distinctive
API_KEY	≥ 0.90	≥ 0.85	≥ 0.87	≤ 0.05	Generic high-entropy patterns have inherently higher FPR
EMAIL	≥ 0.97	≥ 0.95	≥ 0.96	≤ 0.01	RFC 5321 patterns are well-defined
PHONE	≥ 0.92	≥ 0.88	≥ 0.90	≤ 0.03	International formats increase ambiguity
IBAN	≥ 0.98	≥ 0.95	≥ 0.96	≤ 0.01	IBAN checksum validation enables high precision
PRIVATE_KEY	≥ 0.99	≥ 0.99	≥ 0.99	≤ 0.001	PEM headers are highly distinctive
JWT	≥ 0.98	≥ 0.97	≥ 0.97	≤ 0.005	Three-segment base64url structure is distinctive

When thresholds are breached

A threshold breach surfaces in three ways depending on configuration:

1. Hard fail (default for precision and FPR): CI marks the check as failed and blocks the merge. The PR author must investigate and resolve before the branch can land.

2. Soft fail (default for recall and F1): CI marks the check as a warning. The PR can merge, but the breach is recorded in the accuracy report and must be addressed in a follow-up issue within the current sprint.

3. Threshold override: For specific entity types with known structural limitations, you can configure a lower threshold in accuracy.config.json. Overrides require justification comments in the config file and reviewer sign-off.

---

Confidence calibration

What calibration means

A detection confidence score of 0.85 should mean “this finding is correct roughly 85% of the time.” When confidence scores track actual accuracy in this way, the pipeline is said to be well-calibrated. Good calibration enables operators to set meaningful confidence thresholds — choosing where to place the REDACT/BLOCK action boundary based on known precision at each confidence level.

A poorly calibrated pipeline has confidence scores that do not correspond to actual accuracy. A model that always outputs 0.95 confidence regardless of actual accuracy is useless as a threshold tuning tool even if its binary accuracy is acceptable.

Calibration curves

The harness produces a calibration curve for each tier that produces confidence scores. A calibration curve plots:

• X axis: predicted confidence (bucketed into 0.1-wide intervals)
• Y axis: actual fraction of findings in that bucket that were correct (actual precision)

A perfectly calibrated pipeline produces a straight diagonal line. Real pipelines deviate from this ideal, and the shape of the deviation is informative.

Bucket analysis

The harness produces a bucket analysis table showing, for each confidence interval, the number of samples, the number of TPs, the number of FPs, and the actual precision:

Tier 3 Confidence Calibration (CREDIT_CARD, n=240 samples)
─────────────────────────────────────────────────────────────
Bucket      Samples   TPs   FPs   Actual Precision
0.50–0.60       12     9     3           0.750
0.60–0.70       18    14     4           0.778
0.70–0.80       31    26     5           0.839
0.80–0.90       44    41     3           0.932
0.90–0.95       62    60     2           0.968
0.95–1.00       73    72     1           0.986
─────────────────────────────────────────────
Total          240   222    18           0.925

This analysis tells you exactly what precision to expect at each confidence level, which informs your policy threshold configuration. If your policy is configured to BLOCK at confidence ≥ 0.90, the calibration table tells you that you are accepting ~3.2% FP rate in block decisions for this entity type.

Expected behavior per tier

Tier	Expected calibration behavior
Tier 1 (Regex)	Confidence is always 1.0. No calibration curve is produced. Precision at the 1.0 bucket is the only relevant figure.
Tier 2 (NER)	Confidence scores are ordinal, not probabilistic. The calibration curve may be non-monotonic. Use bucket analysis for threshold guidance, not the curve itself.
Tier 3 (DeBERTa)	Well-calibrated. The calibration curve should closely track the diagonal. Deviations indicate model drift or distribution shift in the corpus.
Tier 4 (CredInt)	Confidence is not applicable in the traditional sense. CredInt returns a frequency bucket (LOW / MEDIUM / HIGH / CRITICAL) rather than a continuous confidence score. The harness maps these to ordinal values for reporting.

---

Running the accuracy suite

Prerequisites

Before running the accuracy suite, verify the following:

1. The corpus file exists and is valid:

   npm run dlp:corpus:validate

2. The platform DLP service is running and accessible. The harness sends real HTTP requests to the pipeline — it does not stub the service.

3. If evaluating Tier 4, the CredInt microservice is running:

   curl -s http://credint:8202/health | jq .status
   # Expected: "ok"

4. The baseline file exists (required for regression comparison):

   ls tests/dlp/baselines/current.json

Full suite run

npm run dlp:accuracy

Example output:

Arbitex DLP Accuracy Suite — 2026-03-14T09:15:32Z
Corpus: tests/dlp/corpus/golden.jsonl (1,847 samples)
Pipeline: http://localhost:8300 (outpost) / http://platform:8200 (platform)
Baseline: tests/dlp/baselines/current.json (2026-02-28)

Running Tier 1 evaluation...  done (312 samples, 4.2s)
Running Tier 2 evaluation...  done (418 samples, 23.1s)
Running Tier 3 evaluation...  done (891 samples, 142.8s)
Running Tier 4 evaluation...  done (226 samples, 8.6s)

─────────────────────────────────────────────────────────────────────────
AGGREGATE RESULTS
─────────────────────────────────────────────────────────────────────────
Overall Precision : 0.967  (threshold: ≥ 0.95)  PASS
Overall Recall    : 0.928  (threshold: ≥ 0.90)  PASS
Overall F1        : 0.947  (threshold: ≥ 0.92)  PASS
Overall FPR       : 0.011  (threshold: ≤ 0.02)  PASS

Latency (p50/p95/p99): 4.1ms / 148.2ms / 156.7ms
Tier 3 invocation rate: 48.2%
Tier 4 hit rate: 3.1%

─────────────────────────────────────────────────────────────────────────
PER-ENTITY RESULTS
─────────────────────────────────────────────────────────────────────────
Entity          Prec    Recall  F1      FPR     Status
CREDIT_CARD     0.992   0.964   0.978   0.004   PASS
SSN             0.961   0.937   0.949   0.018   PASS
AWS_KEY         0.998   0.991   0.994   0.002   PASS
API_KEY         0.912   0.877   0.894   0.041   PASS
EMAIL           0.981   0.967   0.974   0.007   PASS
PHONE           0.934   0.898   0.916   0.024   PASS
IBAN            0.987   0.956   0.971   0.006   PASS
BEARER_TOKEN    0.974   0.943   0.958   0.009   PASS
JWT             0.991   0.988   0.989   0.003   PASS
PRIVATE_KEY     0.999   0.997   0.998   0.001   PASS

─────────────────────────────────────────────────────────────────────────
BASELINE COMPARISON
─────────────────────────────────────────────────────────────────────────
Metric          Current  Baseline  Delta
Overall Prec    0.967    0.961     +0.006  improved
Overall Recall  0.928    0.924     +0.004  improved
Overall F1      0.947    0.942     +0.005  improved
Overall FPR     0.011    0.014     -0.003  improved

No regressions detected.

Full report: tests/dlp/results/latest.json

Tip

The example output above is illustrative of the format. Actual numbers will be populated after platform-0076 ships the evaluation harness and corpus.

Running against specific entity types or tiers

# Entity type only
npm run dlp:accuracy -- --entity-type SSN
npm run dlp:accuracy -- --entity-type CREDIT_CARD --entity-type IBAN

# Tier only
npm run dlp:accuracy -- --tier 1
npm run dlp:accuracy -- --tier 3

# Both (only Tier 2 samples for PHONE)
npm run dlp:accuracy -- --tier 2 --entity-type PHONE

# Skip Tier 4 (useful when CredInt is unavailable)
DLP_ACCURACY_SKIP_TIER4=true npm run dlp:accuracy

# Verbose per-sample output
npm run dlp:accuracy -- --verbose 2>&1 | head -100

Output format

The harness writes a structured JSON report to tests/dlp/results/latest.json. The top-level structure is:

{
  "run_id": "acc-20260314-091532",
  "timestamp": "2026-03-14T09:15:32Z",
  "corpus_file": "tests/dlp/corpus/golden.jsonl",
  "corpus_samples": 1847,
  "pipeline_version": "1.14.2",
  "aggregate": {
    "precision": 0.967,
    "recall": 0.928,
    "f1": 0.947,
    "fpr": 0.011,
    "fnr": 0.072,
    "latency_p50_ms": 4.1,
    "latency_p95_ms": 148.2,
    "latency_p99_ms": 156.7,
    "tier3_invocation_rate": 0.482,
    "tier4_hit_rate": 0.031
  },
  "by_entity": {
    "CREDIT_CARD": {
      "tp": 184,
      "fp": 2,
      "fn": 7,
      "tn": 52,
      "precision": 0.992,
      "recall": 0.964,
      "f1": 0.978,
      "fpr": 0.004,
      "threshold_status": "pass"
    }
  },
  "by_tier": {
    "1": {
      "samples": 312,
      "precision": 0.989,
      "recall": 0.954,
      "f1": 0.971
    }
  },
  "calibration": {
    "tier_3": {
      "buckets": [
        { "min": 0.50, "max": 0.60, "samples": 12, "tp": 9, "fp": 3, "actual_precision": 0.750 }
      ]
    }
  },
  "baseline_comparison": {
    "baseline_file": "tests/dlp/baselines/current.json",
    "baseline_date": "2026-02-28",
    "regressions": [],
    "improvements": ["precision", "recall", "f1", "fpr"]
  },
  "samples": []
}

When --verbose is passed, the samples array contains one entry per corpus sample with the actual pipeline response, the expected response, the outcome (TP/FP/FN/TN), and latency.

---

Interpreting results

Reading the summary report

The summary output groups results into four sections:

1. Aggregate results: overall metrics for the full corpus run. Check these first — a global regression here means something changed across the pipeline broadly.

2. Per-entity results: metrics for each entity type with pass/fail status against configured thresholds. A failure here is scoped to one entity type and points to a specific problem area.

3. Baseline comparison: delta against the stored baseline. Improvements and regressions are both highlighted. No regressions is the normal expected outcome for non-breaking pipeline changes.

4. Calibration summary: confidence bucket analysis for Tier 3 and a flag if calibration has drifted from the previous baseline.

Identifying problem areas

Low recall on a specific entity type means the pipeline is missing positive samples. Possible causes:

• A new format variant not covered by existing regex patterns
• A NER model that does not generalize to the sample distribution
• Tier 3 confidence threshold set too high, causing true positives to be dropped

Low precision on a specific entity type means the pipeline is flagging benign content. Possible causes:

• A regex pattern that is too broad (no checksum validation, overly permissive character class)
• NER model producing false positives on common English tokens
• Tier 3 confidence threshold set too low

High FPR on a specific entity type is the most operationally impactful problem. Every false positive on that entity type means a user is blocked or their content is redacted incorrectly.

Drill-down: finding failing samples

Run the harness with --verbose and filter the JSON output to find the specific failing samples:

# Run with verbose output saved to file
npm run dlp:accuracy -- --entity-type API_KEY --verbose \
  --output tests/dlp/results/api-key-debug.json

# Find false positives (entity flagged but shouldn't be)
jq '.samples[] | select(.outcome == "FP")' \
  tests/dlp/results/api-key-debug.json

# Find false negatives (entity missed)
jq '.samples[] | select(.outcome == "FN")' \
  tests/dlp/results/api-key-debug.json

# Find samples that hit Tier 3 with low confidence
jq '.samples[] | select(.tier == 3 and .confidence < 0.70)' \
  tests/dlp/results/api-key-debug.json

Example: investigating a low-precision entity type

Suppose the accuracy suite shows API_KEY precision at 0.882, below the 0.90 threshold. The investigation workflow is:

1. Run verbose mode for API_KEY:

   npm run dlp:accuracy -- --entity-type API_KEY --verbose \
     --output tests/dlp/results/api-key-debug.json

2. Inspect false positives:

   jq '.samples[] | select(.outcome == "FP") | {id, text, tier, confidence, notes}' \
     tests/dlp/results/api-key-debug.json

3. Examine the output. You might find:

   {
     "id": "api-key-neg-047",
     "text": "Set STRIPE_PUB_KEY=pk_test_abcdefghijk for test mode",
     "tier": 1,
     "confidence": 1.0,
     "notes": "Stripe publishable test key, non-sensitive, public prefix"
   }

4. The Stripe publishable key format (pk_test_...) is triggering the generic API key pattern. Since publishable keys are intentionally public, this is a false positive.

5. Remediation options:

   • Add a Tier 1 exception for the pk_test_ and pk_live_ prefixes in the regex
   • Add a Tier 3 hypothesis that explicitly marks publishable key contexts as not sensitive
   • Add the corpus sample to the negative set with a note explaining the classification

6. After remediation, re-run and verify precision improves without recall regression.

---

Updating baselines

When to update

Update the accuracy baseline when you have intentionally changed the pipeline in a way that affects detection behavior and the new behavior is correct. Examples:

• Adding a new regex pattern for a previously uncovered format (recall improves)
• Tightening an overly broad pattern to eliminate a known FP category (precision improves, recall may decrease slightly)
• Updating the NER model to a new spaCy version
• Adjusting a Tier 3 confidence threshold

Do not update the baseline to paper over a regression you do not understand. If metrics degrade and you cannot explain why, investigate before updating.

Baseline file format

The baseline is a JSON file at tests/dlp/baselines/current.json:

{
  "baseline_date": "2026-02-28",
  "baseline_run_id": "acc-20260228-143211",
  "pipeline_version": "1.13.8",
  "aggregate": {
    "precision": 0.961,
    "recall": 0.924,
    "f1": 0.942,
    "fpr": 0.014
  },
  "by_entity": {
    "CREDIT_CARD": {
      "precision": 0.989,
      "recall": 0.959,
      "f1": 0.974,
      "fpr": 0.005
    }
  }
}

Historical baselines are archived in tests/dlp/baselines/history/ with filenames of the form baseline-{YYYY-MM-DD}-{run_id}.json.

Updating the baseline

# Review what changed before committing the update
npm run dlp:accuracy -- --output tests/dlp/results/latest.json
npm run dlp:accuracy:diff  # shows delta from current baseline

# If the diff looks correct, update the baseline
npm run dlp:accuracy:update-baseline

# This copies tests/dlp/results/latest.json to
# tests/dlp/baselines/current.json and archives the old baseline

The update-baseline command will:

1. Run the full accuracy suite to ensure you are updating to a complete and valid result.
2. Copy the current current.json to history/ with a timestamped filename.
3. Write the new baseline to current.json.
4. Print a diff summary showing what changed.

Reviewing baseline diffs before committing

After running update-baseline, review the diff before staging:

git diff tests/dlp/baselines/current.json

A normal baseline update diff shows metric changes that are small, directionally positive, and consistent with the pipeline change being made. Red flags:

• Large unexplained drops in recall or F1
• Changes to entity types that were not affected by your pipeline change
• Aggregate FPR increasing significantly

If you see red flags, revert the baseline update and investigate:

git checkout tests/dlp/baselines/current.json

Approval workflow

Baseline updates require reviewer sign-off. The PR template includes:

- [ ] Accuracy baseline updated (if pipeline behavior changed)
- [ ] Baseline diff reviewed and changes are explained
- [ ] No unexplained regressions in the baseline diff

If the baseline diff shows a regression on any entity type — even if the aggregate metrics pass — the PR description must explain why the regression is acceptable (for example, a known trade-off between precision and recall for a specific entity type, with a follow-up issue filed).

---

CI regression gates

How accuracy tests run in CI

The accuracy suite runs as part of the ci-dlp-accuracy GitHub Actions workflow, triggered on:

• Every push to a pull request branch that modifies files under src/arbitex/dlp/, tests/dlp/, or docker/platform/
• Every push to main
• Scheduled: daily at 02:00 UTC against main

The CI workflow:

1. Spins up the platform service and, optionally, the CredInt microservice in a test network
2. Loads the corpus from tests/dlp/corpus/golden.jsonl
3. Runs the full accuracy suite
4. Compares results against tests/dlp/baselines/current.json
5. Checks each metric against thresholds in accuracy.config.json
6. Reports pass/fail per metric, per entity type

Gate configuration

Thresholds and gate behavior are configured in tests/dlp/accuracy.config.json:

{
  "global_thresholds": {
    "precision": { "min": 0.95, "gate": "hard" },
    "recall": { "min": 0.90, "gate": "soft" },
    "f1": { "min": 0.92, "gate": "soft" },
    "fpr": { "max": 0.02, "gate": "hard" }
  },
  "entity_thresholds": {
    "CREDIT_CARD": {
      "precision": { "min": 0.98, "gate": "hard" },
      "recall": { "min": 0.95, "gate": "soft" },
      "fpr": { "max": 0.01, "gate": "hard" }
    },
    "API_KEY": {
      "precision": { "min": 0.90, "gate": "soft" },
      "recall": { "min": 0.85, "gate": "soft" },
      "fpr": { "max": 0.05, "gate": "soft" },
      "_comment": "API_KEY has inherently higher ambiguity; soft gates only"
    }
  },
  "regression_gate": {
    "enabled": true,
    "max_regression_precision": 0.02,
    "max_regression_recall": 0.03,
    "gate": "hard"
  }
}

Gate modes

Mode	Behavior	Use case
hard	CI fails, merge is blocked	Precision and FPR — direct user impact
soft	CI warns, merge is allowed, issue must be filed	Recall and F1 — important but rarely blocking

Regression gate

The regression gate is separate from the threshold gate. Even if all absolute thresholds pass, the regression gate fires if any metric has decreased by more than the configured maximum regression delta since the last baseline. This catches gradual drift that stays above the absolute floor.

The regression gate is hard by default. If a pipeline change intentionally decreases recall (for example, tightening a noisy pattern to improve precision), you must update the baseline before the PR can land.

Failure modes and example CI output

Hard fail — precision below threshold:

✗ HARD FAIL: CREDIT_CARD precision 0.973 < threshold 0.980
  TP: 179, FP: 5, FN: 7, TN: 54
  Failing samples:
    cc-neg-031: "Product code CC-4532015112830366 (internal ID)" → FP (Tier 1)
    cc-neg-044: "Test: 4532 0151 1283 0366 (Luhn test vector)" → FP (Tier 1)
    ...

Action required: Investigate false positives before merging.

Soft fail — recall below threshold:

⚠ SOFT WARN: PHONE recall 0.877 < threshold 0.880
  TP: 308, FP: 14, FN: 43, TN: 189

  Merge is allowed. File a follow-up issue: PHONE recall regression (delta -0.011).

Regression gate fail:

✗ HARD FAIL: Regression gate — API_KEY precision regressed by 0.031 (limit: 0.020)
  Current:  0.891
  Baseline: 0.922

  If this regression is intentional, update the accuracy baseline before merging.
  Run: npm run dlp:accuracy:update-baseline

Threshold overrides for specific entity types

If an entity type has a known structural limitation that prevents it from meeting the global threshold, you can configure an entity-specific override in accuracy.config.json. All overrides require:

1. A _comment field in the config explaining the justification.
2. A referenced issue number where the limitation is tracked.
3. Reviewer sign-off during the PR that introduced the override.

"NPI": {
  "precision": { "min": 0.88, "gate": "hard" },
  "_comment": "NPI 10-digit format overlaps with phone numbers in international formats. Tracked in platform-0088.",
  "_issue": "platform-0088"
}

Overrides should be revisited when the referenced issue is resolved.

---

Isolated vs integrated testing

Accuracy measurement operates in two modes that answer different questions about pipeline performance.

Isolated tier testing

Isolated testing evaluates each detection tier independently, with the other tiers disabled. This isolates each tier’s contribution and identifies where specific detection problems originate.

# Test Tier 1 (regex) in isolation
npm run dlp:accuracy -- --tier 1 --isolated

# Test Tier 2 (NER) in isolation
npm run dlp:accuracy -- --tier 2 --isolated

# Test Tier 3 (DeBERTa) in isolation — requires Tier 1/2 candidates
npm run dlp:accuracy -- --tier 3 --isolated

Isolated testing reveals:

• Tier 1 precision floor: with only regex, what is the baseline false positive rate? This is the FP rate your users experience on entities where regex is the sole detector.
• Tier 2 recall ceiling: without regex pre-filtering, how many entities does NER catch independently? This measures NER’s standalone detection capability.
• Tier 3 precision lift: comparing Tier 1+2 precision with and without Tier 3 validation quantifies DeBERTa’s contribution to false positive reduction.

Tip

When investigating a precision regression on a specific entity type, run isolated Tier 1 testing first. If Tier 1 precision is acceptable but integrated precision is low, the problem is in Tier 2 (NER producing FPs that Tier 3 fails to filter). If Tier 1 precision is already low, the regex pattern itself is too broad.

Integrated pipeline testing

Integrated testing runs all tiers together — the same configuration that operates in production. This measures the pipeline’s actual end-to-end performance including cross-tier interactions: Tier 3 demoting Tier 1 false positives, Tier 2 catching entities that Tier 1 misses, deduplication across tiers.

# Full integrated pipeline test (default mode)
npm run dlp:accuracy

Integrated results are the operationally relevant metrics — they predict what users will experience. Isolated results are diagnostic — they explain why integrated results look the way they do.

When to use each mode

Scenario	Mode	Why
CI regression gate	Integrated	Measures production behavior
Investigating a precision drop	Isolated (per-tier)	Identifies which tier introduced the regression
Adding a new regex pattern	Isolated (Tier 1) + Integrated	Verify the pattern in isolation, then confirm no cross-tier interference
Tuning DeBERTa confidence threshold	Integrated	Tier 3 only operates on Tier 1+2 candidates in integrated mode
Evaluating a new NER model	Isolated (Tier 2) + Integrated	Measure standalone NER improvement, then confirm integrated improvement

---

Shadowing analysis

Shadowing measures how the pipeline performs against live production traffic without affecting request handling. The DLP pipeline runs in shadow mode on a configurable percentage of traffic, producing detection results that are logged but not enforced.

How shadowing works

1. A configurable traffic sample (default: 5%) is duplicated to the shadow pipeline
2. The shadow pipeline runs the full 5-tier detection stack
3. Shadow results are logged with mode: "shadow" in the audit trail
4. Shadow detections do not trigger actions (no redaction, no blocking, no user-visible effects)
5. Shadow results are compared against the production pipeline’s results for the same requests

Shadow analysis metrics

The shadow analysis produces two classes of metrics:

Agreement metrics — how often the shadow pipeline and production pipeline agree:

Metric	Definition
Agreement rate	Fraction of requests where shadow and production produce identical entity sets
Shadow-only detections	Entities found by shadow but not production (indicates recall improvement in shadow config)
Production-only detections	Entities found by production but not shadow (indicates recall regression in shadow config)
Action agreement	Fraction of requests where shadow and production would take the same action

Shadow-specific accuracy — when shadow detections are manually reviewed:

Metric	Definition
Shadow precision	Fraction of shadow-only detections that are true positives (manual review required)
Shadow recall lift	Additional recall from shadow-only detections, expressed as percentage points above production recall

Running shadow analysis

# Analyze shadow logs from the last 24 hours
npm run dlp:shadow:analyze -- --hours 24

# Analyze shadow logs for a specific entity type
npm run dlp:shadow:analyze -- --entity-type API_KEY --hours 168

# Export shadow analysis to JSON
npm run dlp:shadow:analyze -- --output tests/dlp/results/shadow-report.json

Using shadow results to validate configuration changes

Before deploying a pipeline configuration change (new pattern, threshold adjustment, model update), run the new configuration in shadow mode for a minimum of 48 hours against production traffic. Compare the shadow results to the current production results:

1. If shadow precision ≥ production precision and shadow recall ≥ production recall: the change is safe to promote.
2. If shadow precision < production precision: the change introduces false positives. Investigate before promoting.
3. If shadow recall < production recall: the change drops entities. Investigate whether the dropped entities are true positives (regression) or false positives being correctly filtered (improvement).

---

Confidence calibration findings

Phase 2 analysis of the DeBERTa confidence calibration revealed several patterns that affect how confidence thresholds should be interpreted and configured.

Calibration by entity type

DeBERTa confidence calibration varies significantly across entity types. Some entity types produce well-calibrated scores (predicted confidence tracks actual precision closely), while others show systematic overconfidence or underconfidence.

Entity type	Calibration quality	Finding
CREDIT_CARD	Excellent	Confidence tracks precision within ±2% across all buckets. Luhn validation provides a strong structural signal.
SSN	Good	Slight overconfidence in the 0.70–0.80 range (predicted 0.75, actual 0.71). Nine-digit patterns are inherently ambiguous.
API_KEY	Moderate	Systematic overconfidence above 0.80 (predicted 0.90, actual 0.83). High-entropy strings are difficult to classify contextually.
EMAIL	Excellent	RFC 5321 format provides strong signal. Calibration within ±1%.
PHONE	Fair	Underconfident on international formats (predicted 0.65, actual 0.78). The model is appropriately cautious on ambiguous number sequences.
PERSON	Fair	Context-dependent. Well-calibrated in formal contexts (“Patient: John Smith”), overconfident in informal contexts (“tell john about it”).
IBAN	Excellent	Checksum validation makes calibration trivial.
PRIVATE_KEY	Excellent	PEM header format is unambiguous.

Implications for threshold configuration

Based on the calibration findings:

• Entity types with excellent calibration (CREDIT_CARD, EMAIL, IBAN, PRIVATE_KEY): confidence thresholds directly predict the false positive rate. Setting the action boundary at 0.90 confidence means approximately 3% of detections at that level are false positives.
• Entity types with moderate calibration (API_KEY, PHONE): confidence thresholds are approximate. A 0.90 threshold may produce 5–8% false positives rather than the expected 3%. Use the bucket analysis table for that entity type rather than relying on the raw confidence value.
• Entity types with fair calibration (PERSON): consider setting a higher threshold (0.85+) to compensate for overconfidence, or rely on Tier 3 DeBERTa validation to filter false positives.

---

Pattern fix methodology

When the accuracy suite identifies a precision or recall problem, this methodology provides a systematic approach to diagnosis and resolution.

Step 1: Classify the failure mode

Failure	Meaning	Investigation
False positive (FP)	Benign text flagged as sensitive	Examine the text — is the pattern too broad? Is checksum validation missing?
False negative (FN)	Sensitive text not flagged	Examine the text — is the format variant covered by an existing pattern? Is NER failing to provide context?
Confidence error	Correct detection but wrong confidence	Examine calibration — is DeBERTa scoring appropriately for this context?

Step 2: Isolate the tier

Run the accuracy suite in isolated mode for each tier to determine which tier is responsible:

# If the problem is a false positive:
npm run dlp:accuracy -- --tier 1 --isolated --entity-type API_KEY --verbose
# Check: does Tier 1 produce the FP, or does Tier 2?

# If the problem is a false negative:
npm run dlp:accuracy -- --tier 2 --isolated --entity-type PHONE --verbose
# Check: does Tier 2 catch it independently?

Step 3: Apply the fix

Problem	Fix approach
Tier 1 FP: pattern too broad	Narrow the regex character class, add checksum validation, or add a negative lookahead
Tier 1 FN: format not covered	Add a new pattern variant or relax the existing pattern (verify precision does not degrade)
Tier 2 FP: NER overdetects	Add the false-positive text pattern to the corpus as a negative sample; consider a Tier 3 hypothesis to filter it
Tier 2 FN: NER misses context	Add positive corpus samples with the missed format; if systematic, consider a custom Presidio recognizer
Tier 3 over-demotes	Lower the DeBERTa confidence threshold for the entity type, or add positive training examples
Tier 3 under-demotes	Raise the confidence threshold, or add negative training examples that match the FP pattern

Step 4: Verify and baseline

After applying the fix:

# Run the full suite to verify the fix
npm run dlp:accuracy

# Check for regressions across all entity types
npm run dlp:accuracy:diff

# If the diff shows improvement with no regressions, update the baseline
npm run dlp:accuracy:update-baseline

---

Accuracy target thresholds

The following thresholds define minimum acceptable accuracy for each entity type. These serve as CI regression gates and as benchmarks for evaluating pipeline changes.

Tier-specific targets

Tier	Metric	Target	Rationale
Tier 1 (Regex)	Precision	≥ 0.97	Regex with checksum validation should rarely fire on benign text
Tier 1 (Regex)	Recall	≥ 0.85	Bounded by format coverage — not all variants can be captured by regex
Tier 2 (NER)	Precision	≥ 0.88	ML-based detection has inherent FP rate from ambiguous text
Tier 2 (NER)	Recall	≥ 0.80	NER complements regex for context-dependent entities
Tier 3 (DeBERTa)	Precision lift	≥ +3%	DeBERTa must measurably improve precision over Tier 1+2 alone
Tier 3 (DeBERTa)	Recall preservation	≥ -1%	DeBERTa should not significantly reduce recall through over-filtering
Tier 4 (CredInt)	Precision	≥ 0.9999	Bloom filter FPR determines precision floor
Tier 4 (CredInt)	Recall	N/A	Bounded by corpus coverage, not a tunable parameter

Integrated pipeline targets (production-equivalent)

Entity type	Precision	Recall	F1	FPR	Risk tier
CREDIT_CARD	≥ 0.98	≥ 0.95	≥ 0.96	≤ 0.01	Critical — financial data
SSN	≥ 0.95	≥ 0.92	≥ 0.93	≤ 0.02	Critical — government ID
AWS_KEY	≥ 0.99	≥ 0.98	≥ 0.98	≤ 0.005	Critical — credential
API_KEY	≥ 0.90	≥ 0.85	≥ 0.87	≤ 0.05	High — generic credential
EMAIL	≥ 0.97	≥ 0.95	≥ 0.96	≤ 0.01	Medium — structured PII
PHONE	≥ 0.92	≥ 0.88	≥ 0.90	≤ 0.03	Medium — format variability
IBAN	≥ 0.98	≥ 0.95	≥ 0.96	≤ 0.01	Critical — financial data
PRIVATE_KEY	≥ 0.99	≥ 0.99	≥ 0.99	≤ 0.001	Critical — credential
JWT	≥ 0.98	≥ 0.97	≥ 0.97	≤ 0.005	Critical — credential
BEARER_TOKEN	≥ 0.96	≥ 0.93	≥ 0.94	≤ 0.01	High — credential
PERSON	≥ 0.85	≥ 0.80	≥ 0.82	≤ 0.08	Medium — context-dependent
PASSPORT	≥ 0.94	≥ 0.90	≥ 0.92	≤ 0.02	Critical — government ID

Threshold rationale

Thresholds are set based on three factors:

1. Detection difficulty: entity types with distinctive structural features (PEM headers, Luhn-valid numbers, AKIA prefixes) can achieve higher precision. Entity types with ambiguous formats (phone numbers, generic API keys) have inherently lower precision ceilings.

2. Risk profile: high-risk entity types (credentials, financial data, government IDs) have stricter thresholds because the cost of a false negative (data leak) outweighs the cost of a false positive (user inconvenience).

3. Operational impact: entity types with high traffic volume (email, phone) have moderate FPR thresholds because even a low FPR at scale generates significant noise. Entity types with low traffic volume (PRIVATE_KEY, PASSPORT) can tolerate stricter FPR gates because the absolute number of false positives is small.

---

How to interpret accuracy reports

The accuracy report contains multiple sections that answer different operational questions. This guide explains what each section means and what actions to take based on the findings.

Reading the aggregate summary

The aggregate summary at the top of the report shows pipeline-wide metrics:

Overall Precision : 0.967  (threshold: ≥ 0.95)  PASS
Overall Recall    : 0.928  (threshold: ≥ 0.90)  PASS
Overall F1        : 0.947  (threshold: ≥ 0.92)  PASS
Overall FPR       : 0.011  (threshold: ≤ 0.02)  PASS

What to check first: if all four metrics pass, the pipeline is performing within acceptable bounds. Move to per-entity results to check for entity-specific problems that are masked by strong overall numbers.

If aggregate precision fails: the pipeline is flagging too much benign content. This directly affects user experience — users see incorrect redactions or blocks. Priority: high. Check the per-entity breakdown to identify which entity type is contributing the most false positives.

If aggregate recall fails: the pipeline is missing sensitive content. This is a security concern — data may be reaching model providers undetected. Priority: high for critical entity types (credentials, financial data), medium for others.

If aggregate F1 fails but precision and recall individually pass: this indicates an imbalance. One of the two metrics is near its threshold while the other is well above. Check whether the near-threshold metric is trending downward.

Reading per-entity results

The per-entity table breaks down metrics by entity type:

Entity          Prec    Recall  F1      FPR     Status
CREDIT_CARD     0.992   0.964   0.978   0.004   PASS
API_KEY         0.912   0.877   0.894   0.041   PASS

Look for entity types near their thresholds. An entity type at 0.91 precision with a 0.90 threshold is one pattern change away from a CI failure. Track these as regression risks.

Look for entity types with high FPR. Even if overall FPR is acceptable, a single entity type with FPR > 0.05 means 1 in 20 negative samples for that type triggers a false detection. If that entity type is configured with BLOCK or REDACT action, users of that entity type experience significant disruption.

Look for entity types with low recall. Recall below 0.85 for a critical entity type (SSN, CREDIT_CARD, AWS_KEY) means the pipeline is missing more than 15% of sensitive content of that type. Run the verbose harness to identify which format variants are being missed.

Reading the baseline comparison

The baseline comparison shows deltas from the last accepted baseline:

Metric          Current  Baseline  Delta
Overall Prec    0.967    0.961     +0.006  improved
Overall Recall  0.928    0.924     +0.004  improved

Green path: all deltas are positive or zero. The pipeline is improving or stable.

Yellow path: a metric decreased by less than the regression gate maximum (typically 0.02 for precision, 0.03 for recall). No CI failure, but investigate whether the decrease is explained by the current change.

Red path: a metric decreased by more than the regression gate maximum. CI fails. You must either fix the regression or update the baseline with a justification.

Reading the calibration analysis

The calibration analysis shows whether confidence scores are trustworthy:

Bucket      Samples   TPs   FPs   Actual Precision
0.90–0.95       62    60     2           0.968
0.95–1.00       73    72     1           0.986

Well-calibrated: actual precision tracks the bucket midpoint (±5%). Confidence thresholds can be used directly to set action boundaries.

Overconfident: actual precision is significantly lower than predicted. A detection with confidence 0.90 is correct only 80% of the time. Consider raising the action threshold for this entity type.

Underconfident: actual precision is higher than predicted. The pipeline is being too cautious — you may be able to lower the threshold and catch more entities without increasing false positives.

Accuracy report checklist

Use this checklist when reviewing an accuracy report:

1. Do all aggregate metrics pass their thresholds?
2. Do all per-entity metrics pass their thresholds?
3. Are any entity types within 2% of their threshold (regression risk)?
4. Does the baseline comparison show any decreases?
5. Are calibration curves stable compared to the previous report?
6. For entity types with actions set to BLOCK or REDACT: is the FPR acceptable for the traffic volume?

If all answers are satisfactory, the pipeline is healthy. If any answer raises a concern, run the verbose harness for the affected entity type and follow the pattern fix methodology.

---

Operational notes

Framework status

The accuracy validation framework was specified in platform-0076 and is operational. The golden corpus, evaluation harness, CI gates, and baseline management are all available. The framework supports both isolated and integrated testing modes, shadow analysis, and confidence calibration analysis.

Related documentation

• DLP Detection Hierarchy — how detection tiers run exhaustively, multi-match behavior, action resolution
• DLP Pipeline Configuration — configure tiers, write custom rules, tune thresholds
• Credential Intelligence — Tier 4 CredInt architecture, corpus, and lookup protocol
• DLP Event Monitoring — monitoring DLP detections in production via the audit log
• DLP Redaction Visibility — how redaction is presented to users and admins
• DLP Regulatory Patterns — regulatory compliance pattern coverage

Accuracy in air-gap deployments

Air-gap deployments run the full DLP pipeline locally without outbound network access. The accuracy suite is designed to run in air-gap environments with one constraint: Tier 4 CredInt evaluation requires the CredInt microservice, which must be deployed separately. If CredInt is not deployed in your air-gap environment, set DLP_ACCURACY_SKIP_TIER4=true.

The corpus, harness, and baseline files are included in the platform distribution and do not require internet access to run.

Corpus governance

The golden corpus is a security-sensitive artifact. It contains real examples of sensitive data patterns — SSNs, credit card numbers, API key formats — used as test vectors. Corpus files are treated as code:

• Committed to version control with the same access controls as source code
• Never included in log output, telemetry, or error reports in plaintext
• Test corpus entries use generated or synthetic values where possible (for example, the IRS publishes official SSN test values; Luhn-valid credit card numbers can be generated deterministically)

Caution

Do not add real customer data or credentials to the test corpus. Use synthetic values, officially published test vectors (IRS, card network test ranges), or values generated by the test corpus tooling.

---

Quick-reference: published accuracy by entity type

A concise summary of Tier 1 (regex) F1 scores for key entity types from the production accuracy harness. Full per-entity tables with precision, recall, and example counts are in DLP Accuracy — Methodology & Results.

Entity type	Tier 1 F1	Notes
credit_card	0.979	Luhn-validated; 2 medium-difficulty format variants not yet covered
ssn	0.992
email	0.906	Regex precision 0.827; Tier 3 DeBERTa lifts integrated F1 to 0.996
iban	1.000	mod-97 checksum validated
npi	1.000	Luhn validated
dea_number	1.000	Checksum validated
aws_access_key_id	0.968	AROA prefix variants not yet matched
github_pat	0.984
pem_private_key	1.000
jwt_token	1.000
ip_address	1.000
phone	1.000

For Tier 3 DeBERTa v4 per-entity F1 scores (35 entity types, 281K examples, 99.48% accuracy), see DLP Accuracy — Methodology & Results — Tier 3.