DLP pipeline — technical reference

This reference covers the internal mechanics of the DLP pipeline: how each tier detects sensitive data, how findings flow through the pipeline, how confidence scores are computed and adjusted, and how to tune thresholds. For the enforcement layer that consumes DLP findings, see Policy Engine overview.

---

Pipeline architecture

flowchart TD
    Input["Text (prompt or response)"]

    subgraph T1["Tier 1 — Pattern Matching (in-process)"]
        R1["RegexDetector\n65+ patterns\nLuhn / IBAN / DEA / NPI checksum"]
    end

    subgraph T2["Tier 2 — NER (GPU microservice :8200)"]
        N1["GLiNER zero-shot NER\nurchade/gliner_medium-v2.1\ncircuit breaker: 3 failures / 60s reset"]
    end

    subgraph T3["Tier 3 — Contextual Validation (GPU microservice :8201)"]
        D1["DeBERTa NLI validator\nmicrosoft/deberta-v3-base\nPOST /validate\ncircuit breaker: 3 failures / 60s reset"]
    end

    Input --> T1
    Input --> T2
    T1 & T2 --> Merge["Merge + deduplicate findings"]
    Merge --> T3
    T3 --> Filter["Apply confidence_threshold filter"]
    Filter --> PE["Policy Engine\nentity_types + entity_confidence_min conditions"]

    style T1 fill:#e8f5e9
    style T2 fill:#fff3e0
    style T3 fill:#fce4ec

Tier 1 and Tier 2 run concurrently — the pipeline does not wait for Tier 1 to complete before starting Tier 2. Tier 3 runs after both tiers have returned results. Findings are deduplicated by span overlap (longest span wins on overlap; equal spans keep the highest confidence) before being passed to Tier 3 for contextual validation.

---

Tier 1 — Pattern matching

The RegexDetector runs all patterns against the full text of the input or output. Patterns are grouped into four modules:

Module	What it detects	Example patterns
dlp_patterns_pii	Credentials and digital identifiers	AWS access keys (AKIA...), Anthropic tokens (sk-ant-...), GitHub PATs, JWT tokens, PEM private keys, SMTP passwords, database URLs
dlp_patterns_financial	Payment and banking data	Credit/debit cards (Visa/MC/Amex/Discover/JCB/UnionPay/Diners), IBAN, SWIFT/BIC, US routing numbers, ABA numbers, ACH pairs, crypto wallet addresses
dlp_patterns_medical	Healthcare identifiers	DEA numbers, NPI (10-digit), UK NHS numbers, Canadian health card numbers
dlp_patterns (core)	Broad PII and secrets	SSN, ITIN, EIN, passport numbers, phone numbers, email addresses, IP addresses, driver license formats, Indian PAN/Aadhaar, Australian TFN

65+ patterns total across all modules.

Checksum validators

For structured identifiers where a regex match alone produces false positives, Tier 1 includes validators that verify structural correctness:

Entity type	Validator	Algorithm
credit_card	Luhn check	Luhn mod-10 — validates that the digit string passes the standard card checksum
bank_account_number	IBAN check	IBAN mod-97 — rearranges country code + check digits and verifies remainder = 1
dea_number	DEA check	DEA registration number checksum — weighted digit sum of positions 1/3/5 + 2× sum of positions 2/4/6, check digit is ones-place of total
npi	Luhn check	US NPI number validated using Luhn after prepending ISO prefix 80840
ssn	Format check	Excludes known-invalid SSN ranges (000-XX-XXXX, 666-XX-XXXX, 900–999 area)

When a validator is present for a pattern, the RegexDetector calls it after the regex match. If the validator returns False, the match is discarded — the text matched the format but is not a valid number. This eliminates the majority of false positives for payment card and banking data.

Tier 1 confidence

Tier 1 matches are assigned confidence based on the pattern’s confidence_threshold:

• Secrets (API keys, private keys, connection strings): 0.90–0.95
• Payment card numbers (Luhn-validated): 0.95
• Government IDs (checksum-validated): 0.85–0.90
• Broad PII (phone, email, address patterns): 0.75–0.85

---

Tier 2 — Named entity recognition (NER)

The MicroserviceNERDetector calls the NER GPU microservice at http://ner-gpu:8200 via POST /detect.

Default model: urchade/gliner_medium-v2.1 (zero-shot NER — no task-specific fine-tuning required; entity labels are passed at inference time).

Request format

{
  "text": "Patient Jordan Smith, DOB 1978-06-15, was prescribed...",
  "labels": ["person", "date_of_birth", "health_info", "address"],
  "threshold": 0.5
}

The labels field is the list of canonical entity types from the entity taxonomy. The threshold is set by NER_DEFAULT_THRESHOLD (default 0.5). Lower thresholds increase recall at the cost of more false positives that Tier 3 must filter.

Response format

{
  "entities": [
    {
      "text": "Jordan Smith",
      "label": "person",
      "start": 8,
      "end": 20,
      "score": 0.91
    },
    {
      "text": "1978-06-15",
      "label": "date_of_birth",
      "start": 26,
      "end": 36,
      "score": 0.85
    }
  ],
  "model": "urchade/gliner_medium-v2.1",
  "inference_time_ms": 68
}

Each entity becomes a DLPMatch with the NER score as the initial confidence value.

Circuit breaker

Both the NER and DeBERTa clients share the same circuit breaker pattern:

• Threshold: 3 consecutive HTTP failures, timeouts, or connection errors
• Open window: 60 seconds. During this window, the detector returns empty results (fail-open) without making HTTP calls.
• Half-open probe: After 60 seconds, one request is allowed through. On success, the circuit resets. On failure, the circuit re-opens for another 60 seconds.
• Fail-open guarantee: A degraded or unavailable Tier 2/Tier 3 service never blocks requests. The pipeline continues with whatever findings it has.

NER GPU resource requirements

Resource	Value
Container memory limit	4 GB
GPU requirement	1× NVIDIA GPU (any CUDA-capable)
Model warm-up time	~60 seconds on cold start
Health check endpoint	GET http://ner-gpu:8200/health

If no GPU is available, set DLP_NER_BACKEND=default to fall back to CPU-based Presidio NER (lower accuracy, no GPU requirement).

---

Tier 3 — Contextual validation (DeBERTa NLI)

The DeBERTaValidatorClient calls the DeBERTa microservice at http://deberta-validator:8201 via POST /validate. This tier does not add new findings — it re-evaluates existing findings from Tier 1 and Tier 2 to determine whether they are true or false positives in context.

Default model: microsoft/deberta-v3-base (Natural Language Inference / entailment model).

Purpose: Some entity type matches are ambiguous. The text “Jordan Smith” might appear in:

• A medical record (sensitive — PHI)
• A discussion of a basketball player (not sensitive — common name in context)

Without contextual validation, both produce a name entity finding. DeBERTa uses NLI to classify the relationship between the matched text and its surrounding context, determining whether the match is genuinely sensitive in its specific context.

How DeBERTa validates a finding

For each candidate match, the validator constructs an NLI pair:

• Premise: The surrounding text (context window around the matched span)
• Hypothesis: A sensitivity claim, for example: “This text contains a person’s name in a context where it identifies a real individual.”

The model produces an entailment score. If the score exceeds DEBERTA_THRESHOLD (default 0.70), the match is confirmed as a true positive and its confidence is updated to the entailment score. If below threshold, the match is demoted to confidence 0.1 — below any practical confidence_threshold setting — effectively filtering it out.

Request format

{
  "matches": [
    {
      "text": "Jordan Smith",
      "entity_type": "name",
      "start": 8,
      "end": 20,
      "score": 0.91
    }
  ],
  "context": "Patient Jordan Smith, DOB 1978-06-15, was prescribed Metformin for type 2 diabetes.",
  "threshold": 0.70
}

Response format

{
  "validated_matches": [
    {
      "text": "Jordan Smith",
      "entity_type": "name",
      "start": 8,
      "end": 20,
      "is_true_positive": true,
      "contextual_score": 0.94
    }
  ]
}

The pipeline applies this response by updating each matched finding’s confidence to contextual_score (true positive) or 0.1 (false positive). Findings not present in the response are left unchanged.

DeBERTa GPU resource requirements

Resource	Value
Container memory limit	4 GB
GPU requirement	1× NVIDIA GPU
Max input length	512 tokens (set by DEBERTA_MAX_LENGTH)
Health check endpoint	GET http://deberta-validator:8201/health
Enable/disable	DLP_DEBERTA_ENABLED=true in backend environment

DeBERTa is disabled by default (DLP_DEBERTA_ENABLED=false). Enable it when GPU capacity allows and Tier 1/Tier 2 false positive rates are a concern. When disabled, findings from Tier 1 and Tier 2 pass directly to the Policy Engine at their original confidence scores.

Training status: The DeBERTa model is currently in training (epoch 5 of 6). Accuracy metrics will be published here when training completes. In the meantime, the default threshold of 0.70 is conservative — it will pass most genuine positives while filtering the most obvious false positives.

---

Entity taxonomy

All DLP findings are normalized to a canonical entity type vocabulary before reaching the Policy Engine. This ensures that PERSON (from Presidio), person (from GLiNER), and person_name (from a custom rule) all appear as name in the finding, and that policy rules written against name match regardless of which detector produced the finding.

Categories

Category	Entities
direct_identifiers	name, address, telephone, email, username, fax
employment_financial	employment_info, account_balances, contract_numbers, policy_numbers, financial_account_numbers
digital_location	ip_address, geolocation, website_tracking
government_ids	ssn, drivers_license, passport, military_id, date_of_birth, itin, ein, npi, dea_number, uk_nino, uk_nhs_number, indian_pan, indian_aadhaar, au_tfn, au_medicare, canadian_sin
financial_instruments	bank_account_number, ach_data, credit_card, debit_card, pci_data, cvv, uk_sort_code, swift_bic, crypto_wallet, magstripe, credit_report
protected_health_sensitive	health_info, biometric, genetic, criminal_convictions, racial_ethnic_origin, political_opinions, religious_beliefs, trade_union_membership, sexual_orientation
credentials	api_key, username_password_combo, connection_string, private_key, bearer_token
mnpi	earnings_announcement, merger_acquisition, insider_info, material_contract, regulatory_action, executive_change

62 canonical entity types in total.

Alias resolution

When a detector produces a finding with a non-canonical name (for example, CREDIT_CARD from Presidio, credit_card_number from a custom regex, or credit card from an LLM detector), the pipeline calls normalize_entity_type() to map it to the canonical name credit_card. The lookup is case-insensitive. Unknown names fall through as-is (lowercased) — they can still be referenced in policy rules using the raw name.

---

Confidence scoring

Each finding carries a confidence value (0.0–1.0) that flows through the pipeline:

1. Tier 1: Initial confidence from the pattern’s confidence_threshold (0.75–0.95). Checksum-validated matches get the full value; format-only matches get a lower value.
2. Tier 2: NER detection score from the GLiNER model inference (the score field in the response).
3. Deduplication: When Tier 1 and Tier 2 both detect the same span with different confidences, the higher confidence is kept.
4. Tier 3: DeBERTa updates the confidence to the NLI entailment score for true positives, or demotes to 0.1 for false positives.
5. Threshold filter: Findings below the pipeline’s confidence_threshold (configured per organization) are discarded before results are passed to the Policy Engine.

The finding’s confidence in the Policy Engine represents the final post-validation confidence. The entity_confidence_min condition in a policy rule checks against this final value.

---

Threshold tuning

Three thresholds control DLP sensitivity. Adjust them to balance recall (catching all sensitive data) against precision (avoiding false positives that block legitimate requests).

NER_DEFAULT_THRESHOLD (env var, Tier 2)

Controls GLiNER inference — the minimum score for a detected entity to be returned by the NER microservice.

Value	Effect
0.3	High recall, many false positives. Tier 3 validation is essential at this threshold.
0.5 (default)	Balanced. Tier 3 filters residual false positives.
0.7	High precision, may miss ambiguous entities.

DEBERTA_THRESHOLD (env var, Tier 3)

Controls DeBERTa NLI validation — the minimum entailment score for a finding to be confirmed as a true positive.

Value	Effect
0.5	Permissive. Most findings pass. Useful for high-recall compliance use cases.
0.70 (default)	Conservative. Filters obvious false positives while confirming genuine sensitive data.
0.85	Very selective. Only high-confidence contextual positives survive. Lower false positive rate at the cost of some true positives.

entity_confidence_min (policy rule condition)

A per-rule threshold applied at Policy Engine evaluation time. Set this in individual rules to fine-tune which confidence level of findings triggers enforcement.

{
  "conditions": {
    "entity_types": ["credit_card"],
    "entity_confidence_min": 0.85
  },
  "action": { "type": "BLOCK" }
}

This rule fires only when a credit_card finding has confidence ≥ 0.85 after all three DLP tiers have processed it. Set a lower threshold on audit/log-only rules and a higher threshold on blocking rules to layer detection with graduated responses.

---

Configuration reference

Variable	Default	Service	Description
DLP_NER_BACKEND	default	Backend	NER backend: default (Presidio, CPU), gliner (in-process GLiNER), or microservice (GPU microservice at :8200)
DLP_NER_MICROSERVICE_URL	http://ner-gpu:8200	Backend	NER microservice base URL
DLP_NER_MICROSERVICE_TIMEOUT	5.0	Backend	HTTP timeout (seconds) for NER microservice calls
DLP_DEBERTA_ENABLED	false	Backend	Enable Tier 3 DeBERTa contextual validation
DLP_DEBERTA_VALIDATOR_URL	http://deberta-validator:8201	Backend	DeBERTa microservice base URL
DLP_DEBERTA_VALIDATOR_TIMEOUT	5.0	Backend	HTTP timeout for DeBERTa validation calls
DLP_OUTPUT_SCANNING_ENABLED	true	Backend	Enable DLP scanning on model responses (output) as well as prompts (input)
NER_MODEL_NAME	urchade/gliner_medium-v2.1	NER microservice	GLiNER model identifier
NER_DEFAULT_THRESHOLD	0.5	NER microservice	Default GLiNER inference threshold
DEBERTA_MODEL_NAME	microsoft/deberta-v3-base	DeBERTa microservice	DeBERTa model identifier
DEBERTA_THRESHOLD	0.70	DeBERTa microservice	Minimum entailment score for true positive confirmation
DEBERTA_MAX_LENGTH	512	DeBERTa microservice	Maximum input token length for DeBERTa inference

---

Degraded operation modes

The DLP pipeline is designed to maintain availability under GPU microservice failure:

State	Behavior
Tier 2 (NER) unavailable — circuit open	Pipeline continues with Tier 1 findings only. NER entities (names, addresses, medical terms) not detected until Tier 2 recovers.
Tier 3 (DeBERTa) unavailable — circuit open	Pipeline continues with Tier 1 + Tier 2 findings at their original confidence scores. False positive rate may increase.
Both Tier 2 and Tier 3 unavailable	Pipeline runs Tier 1 only. All checksum-validated structured PII is still detected. Contextual and unstructured PII detection is degraded.
All tiers unavailable	Not possible for Tier 1 — it runs in-process and has no external dependency. Tier 1 is always active.

In all degraded modes, the request is not blocked by the infrastructure failure itself. If a blocking policy rule requires an entity type that is only detected by Tier 2 or Tier 3, those blocks will not fire while those tiers are degraded. Design your policy chain accordingly — use Tier 1 entity types (credit cards, SSNs, API keys) for hard block rules, and Tier 2/3 entity types for advisory or PROMPT rules.

---

See also

• DLP Overview — introduction to the DLP pipeline and how it connects to the Policy Engine
• Policy Rule Reference — entity_types and entity_confidence_min conditions
• Compliance Bundles — compliance framework patterns that run alongside custom DLP rules
• Credential Intelligence — breach corpus checking for credential-type findings