Skip to content

DLP pipeline — technical reference

This reference covers the internal mechanics of the DLP pipeline: how each tier detects sensitive data, how findings flow through the pipeline, how confidence scores are computed and adjusted, and how to tune thresholds. For the enforcement layer that consumes DLP findings, see Policy Engine overview.


flowchart TD
    Input["Text (prompt or response)"]

    subgraph T1["Tier 1 — Pattern Matching (in-process)"]
        R1["RegexDetector\n70+ patterns\nLuhn / IBAN / DEA / NPI checksum"]
    end

    subgraph T2["Tier 2 — NER (GPU microservice :8200)"]
        N1["GLiNER zero-shot NER\nurchade/gliner_medium-v2.1\ncircuit breaker: 3 failures / 60s reset"]
    end

    subgraph T3["Tier 3 — Contextual Validation (GPU microservice :8201)"]
        D1["DeBERTa NLI validator\ndeberta-dlp-v4 (304M params)\nPOST /validate\ncircuit breaker: 3 failures / 60s reset"]
    end

    Input --> T1
    Input --> T2
    T1 & T2 --> Merge["Merge + deduplicate findings"]
    Merge --> T3
    T3 --> Filter["Apply confidence_threshold filter"]
    Filter --> PE["Policy Engine\nentity_types + entity_confidence_min conditions"]

    style T1 fill:#e8f5e9
    style T2 fill:#fff3e0
    style T3 fill:#fce4ec

Tier 1 and Tier 2 run concurrently — the pipeline does not wait for Tier 1 to complete before starting Tier 2. Tier 3 runs after both tiers have returned results. Findings are deduplicated by span overlap (longest span wins on overlap; equal spans keep the highest confidence) before being passed to Tier 3 for contextual validation.


The RegexDetector runs all patterns against the full text of the input or output. Patterns are grouped into four modules:

Module What it detects Example patterns
dlp_patterns_pii Credentials and digital identifiers AWS access keys (AKIA...), Anthropic tokens (sk-ant-...), GitHub PATs, JWT tokens, PEM private keys, SMTP passwords, database URLs
dlp_patterns_financial Payment and banking data Credit/debit cards (Visa/MC/Amex/Discover/JCB/UnionPay/Diners), IBAN, SWIFT/BIC, US routing numbers, ABA numbers, ACH pairs, crypto wallet addresses
dlp_patterns_medical Healthcare identifiers DEA numbers, NPI (10-digit), UK NHS numbers, Canadian health card numbers
dlp_patterns (core) Broad PII and secrets SSN, ITIN, EIN, passport numbers, phone numbers, email addresses, IP addresses, driver license formats, Indian PAN/Aadhaar, Australian TFN

70+ patterns total across all modules.

For structured identifiers where a regex match alone produces false positives, Tier 1 includes validators that verify structural correctness:

Entity type Validator Algorithm
credit_card Luhn check Luhn mod-10 — validates that the digit string passes the standard card checksum
bank_account_number IBAN check IBAN mod-97 — rearranges country code + check digits and verifies remainder = 1
dea_number DEA check DEA registration number checksum — weighted digit sum of positions 1/3/5 + 2× sum of positions 2/4/6, check digit is ones-place of total
npi Luhn check US NPI number validated using Luhn after prepending ISO prefix 80840
ssn Format check Excludes known-invalid SSN ranges (000-XX-XXXX, 666-XX-XXXX, 900–999 area)

When a validator is present for a pattern, the RegexDetector calls it after the regex match. If the validator returns False, the match is discarded — the text matched the format but is not a valid number. This eliminates the majority of false positives for payment card and banking data.

Tier 1 matches are assigned confidence based on the pattern’s confidence_threshold:

  • Secrets (API keys, private keys, connection strings): 0.90–0.95
  • Payment card numbers (Luhn-validated): 0.95
  • Government IDs (checksum-validated): 0.85–0.90
  • Broad PII (phone, email, address patterns): 0.75–0.85

The MicroserviceNERDetector calls the NER GPU microservice at http://ner-gpu:8200 via POST /detect.

Default model: urchade/gliner_medium-v2.1 (zero-shot NER — no task-specific fine-tuning required; entity labels are passed at inference time).

{
"text": "Patient Jordan Smith, DOB 1978-06-15, was prescribed...",
"labels": ["person", "date_of_birth", "health_info", "address"],
"threshold": 0.5
}

The labels field is the list of canonical entity types from the entity taxonomy. The threshold is set by NER_DEFAULT_THRESHOLD (default 0.5). Lower thresholds increase recall at the cost of more false positives that Tier 3 must filter.

{
"entities": [
{
"text": "Jordan Smith",
"label": "person",
"start": 8,
"end": 20,
"score": 0.91
},
{
"text": "1978-06-15",
"label": "date_of_birth",
"start": 26,
"end": 36,
"score": 0.85
}
],
"model": "urchade/gliner_medium-v2.1",
"inference_time_ms": 68
}

Each entity becomes a DLPMatch with the NER score as the initial confidence value.

Both the NER and DeBERTa clients share the same circuit breaker pattern:

  • Threshold: 3 consecutive HTTP failures, timeouts, or connection errors
  • Open window: 60 seconds. During this window, the detector returns empty results (fail-open) without making HTTP calls.
  • Half-open probe: After 60 seconds, one request is allowed through. On success, the circuit resets. On failure, the circuit re-opens for another 60 seconds.
  • Fail-open guarantee: A degraded or unavailable Tier 2/Tier 3 service never blocks requests. The pipeline continues with whatever findings it has.
Resource Value
Container memory limit 4 GB
GPU requirement 1× NVIDIA GPU (any CUDA-capable)
Model warm-up time ~60 seconds on cold start
Health check endpoint GET http://ner-gpu:8200/health

If no GPU is available, set DLP_NER_BACKEND=default to fall back to CPU-based Presidio NER (lower accuracy, no GPU requirement).


Tier 3 — Contextual validation (DeBERTa NLI)

Section titled “Tier 3 — Contextual validation (DeBERTa NLI)”

The DeBERTaValidatorClient calls the DeBERTa microservice at http://deberta-validator:8201 via POST /validate. This tier does not add new findings — it re-evaluates existing findings from Tier 1 and Tier 2 to determine whether they are true or false positives in context.

Default model: DeBERTa-v3-large (304M parameters), fine-tuned as deberta-dlp-v4. Eval accuracy: 99.48% on 281K examples across 35 entity types. See DLP Accuracy Reference for per-entity results.

Purpose: Some entity type matches are ambiguous. The text “Jordan Smith” might appear in:

  • A medical record (sensitive — PHI)
  • A discussion of a basketball player (not sensitive — common name in context)

Without contextual validation, both produce a name entity finding. DeBERTa uses NLI to classify the relationship between the matched text and its surrounding context, determining whether the match is genuinely sensitive in its specific context.

For each candidate match, the validator constructs an NLI pair:

  • Premise: The surrounding text (context window around the matched span)
  • Hypothesis: A sensitivity claim, for example: “This text contains a person’s name in a context where it identifies a real individual.”

The model produces an entailment score. If the score exceeds DEBERTA_THRESHOLD (default 0.70), the match is confirmed as a true positive and its confidence is updated to the entailment score. If below threshold, the match is demoted to confidence 0.1 — below any practical confidence_threshold setting — effectively filtering it out.

{
"matches": [
{
"text": "Jordan Smith",
"entity_type": "name",
"start": 8,
"end": 20,
"score": 0.91
}
],
"context": "Patient Jordan Smith, DOB 1978-06-15, was prescribed Metformin for type 2 diabetes.",
"threshold": 0.70
}
{
"validated_matches": [
{
"text": "Jordan Smith",
"entity_type": "name",
"start": 8,
"end": 20,
"is_true_positive": true,
"contextual_score": 0.94
}
]
}

The pipeline applies this response by updating each matched finding’s confidence to contextual_score (true positive) or 0.1 (false positive). Findings not present in the response are left unchanged.

Resource Value
Container memory limit 4 GB
GPU requirement 1× NVIDIA GPU
Max input length 512 tokens (set by DEBERTA_MAX_LENGTH)
Health check endpoint GET http://deberta-validator:8201/health
Enable/disable DLP_DEBERTA_ENABLED=true in backend environment

DeBERTa is disabled by default (DLP_DEBERTA_ENABLED=false). Enable it when GPU capacity allows and Tier 1/Tier 2 false positive rates are a concern. When disabled, findings from Tier 1 and Tier 2 pass directly to the Policy Engine at their original confidence scores.


All DLP findings are normalized to a canonical entity type vocabulary before reaching the Policy Engine. This ensures that PERSON (from Presidio), person (from GLiNER), and person_name (from a custom rule) all appear as name in the finding, and that policy rules written against name match regardless of which detector produced the finding.

Category Entities
direct_identifiers name, address, telephone, email, username, fax
employment_financial employment_info, account_balances, contract_numbers, policy_numbers, financial_account_numbers
digital_location ip_address, geolocation, website_tracking
government_ids ssn, drivers_license, passport, military_id, date_of_birth, itin, ein, npi, dea_number, uk_nino, uk_nhs_number, indian_pan, indian_aadhaar, au_tfn, au_medicare, canadian_sin
financial_instruments bank_account_number, ach_data, credit_card, debit_card, pci_data, cvv, uk_sort_code, swift_bic, crypto_wallet, magstripe, credit_report
protected_health_sensitive health_info, biometric, genetic, criminal_convictions, racial_ethnic_origin, political_opinions, religious_beliefs, trade_union_membership, sexual_orientation
credentials api_key, username_password_combo, connection_string, private_key, bearer_token
mnpi earnings_announcement, merger_acquisition, insider_info, material_contract, regulatory_action, executive_change

62 canonical entity types in total.

When a detector produces a finding with a non-canonical name (for example, CREDIT_CARD from Presidio, credit_card_number from a custom regex, or credit card from an LLM detector), the pipeline calls normalize_entity_type() to map it to the canonical name credit_card. The lookup is case-insensitive. Unknown names fall through as-is (lowercased) — they can still be referenced in policy rules using the raw name.


Each finding carries a confidence value (0.0–1.0) that flows through the pipeline:

  1. Tier 1: Initial confidence from the pattern’s confidence_threshold (0.75–0.95). Checksum-validated matches get the full value; format-only matches get a lower value.
  2. Tier 2: NER detection score from the GLiNER model inference (the score field in the response).
  3. Deduplication: When Tier 1 and Tier 2 both detect the same span with different confidences, the higher confidence is kept.
  4. Tier 3: DeBERTa updates the confidence to the NLI entailment score for true positives, or demotes to 0.1 for false positives.
  5. Threshold filter: Findings below the pipeline’s confidence_threshold (configured per organization) are discarded before results are passed to the Policy Engine.

The finding’s confidence in the Policy Engine represents the final post-validation confidence. The entity_confidence_min condition in a policy rule checks against this final value.


Three thresholds control DLP sensitivity. Adjust them to balance recall (catching all sensitive data) against precision (avoiding false positives that block legitimate requests).

Controls GLiNER inference — the minimum score for a detected entity to be returned by the NER microservice.

Value Effect
0.3 High recall, many false positives. Tier 3 validation is essential at this threshold.
0.5 (default) Balanced. Tier 3 filters residual false positives.
0.7 High precision, may miss ambiguous entities.

Controls DeBERTa NLI validation — the minimum entailment score for a finding to be confirmed as a true positive.

Value Effect
0.5 Permissive. Most findings pass. Useful for high-recall compliance use cases.
0.70 (default) Conservative. Filters obvious false positives while confirming genuine sensitive data.
0.85 Very selective. Only high-confidence contextual positives survive. Lower false positive rate at the cost of some true positives.

entity_confidence_min (policy rule condition)

Section titled “entity_confidence_min (policy rule condition)”

A per-rule threshold applied at Policy Engine evaluation time. Set this in individual rules to fine-tune which confidence level of findings triggers enforcement.

{
"conditions": {
"entity_types": ["credit_card"],
"entity_confidence_min": 0.85
},
"action": { "type": "BLOCK" }
}

This rule fires only when a credit_card finding has confidence ≥ 0.85 after all three DLP tiers have processed it. Set a lower threshold on audit/log-only rules and a higher threshold on blocking rules to layer detection with graduated responses.


Variable Default Service Description
DLP_NER_BACKEND default Backend NER backend: default (Presidio, CPU), gliner (in-process GLiNER), or microservice (GPU microservice at :8200)
DLP_NER_MICROSERVICE_URL http://ner-gpu:8200 Backend NER microservice base URL
DLP_NER_MICROSERVICE_TIMEOUT 5.0 Backend HTTP timeout (seconds) for NER microservice calls
DLP_DEBERTA_ENABLED false Backend Enable Tier 3 DeBERTa contextual validation
DLP_DEBERTA_VALIDATOR_URL http://deberta-validator:8201 Backend DeBERTa microservice base URL
DLP_DEBERTA_VALIDATOR_TIMEOUT 5.0 Backend HTTP timeout for DeBERTa validation calls
DLP_OUTPUT_SCANNING_ENABLED true Backend Enable DLP scanning on model responses (output) as well as prompts (input)
NER_MODEL_NAME urchade/gliner_medium-v2.1 NER microservice GLiNER model identifier
NER_DEFAULT_THRESHOLD 0.5 NER microservice Default GLiNER inference threshold
DEBERTA_MODEL_NAME deberta-dlp-v4 DeBERTa microservice DeBERTa model identifier (DeBERTa-v3-large, 304M params)
DEBERTA_THRESHOLD 0.70 DeBERTa microservice Minimum entailment score for true positive confirmation
DEBERTA_MAX_LENGTH 512 DeBERTa microservice Maximum input token length for DeBERTa inference

The DLP pipeline is designed to maintain availability under GPU microservice failure:

State Behavior
Tier 2 (NER) unavailable — circuit open Pipeline continues with Tier 1 findings only. NER entities (names, addresses, medical terms) not detected until Tier 2 recovers.
Tier 3 (DeBERTa) unavailable — circuit open Pipeline continues with Tier 1 + Tier 2 findings at their original confidence scores. False positive rate may increase.
Both Tier 2 and Tier 3 unavailable Pipeline runs Tier 1 only. All checksum-validated structured PII is still detected. Contextual and unstructured PII detection is degraded.
All tiers unavailable Not possible for Tier 1 — it runs in-process and has no external dependency. Tier 1 is always active.

In all degraded modes, the request is not blocked by the infrastructure failure itself. If a blocking policy rule requires an entity type that is only detected by Tier 2 or Tier 3, those blocks will not fire while those tiers are degraded. Design your policy chain accordingly — use Tier 1 entity types (credit cards, SSNs, API keys) for hard block rules, and Tier 2/3 entity types for advisory or PROMPT rules.


The pipeline assigns each entity type a default confidence threshold. Combined with the tier thresholds below, this determines the minimum confidence at which a finding survives to reach the Policy Engine.

These thresholds are calibrated from empirical precision measurements (platform-0082, S2). Three action tiers are defined:

Threshold tier Purpose Typical entities
Block tier (≥ 0.70) High-confidence required — false positives here block legitimate requests API keys, private keys, cloud credentials, database connection strings
Redact tier (≥ 0.55) Moderate confidence — matched text is replaced with [REDACTED] SSN, credit card, ITIN, IBAN, passport, NPI, Medicare, driver’s license
Log-only tier (≥ 0.40) Permissive — log for audit without disrupting the user Email, phone number, IP address, MAC address, crypto wallet

The 39 API key and token patterns in the block tier belong to the Arbitex Credential - Secret Key Detection policy pack. See Secret Key Detection — Entity Type Split for the full pack structure, per-entity-type action configuration, and DLP event query migration notes.

Entity type Description
pem_private_key PEM-encoded private key
ssh_private_key SSH RSA/EC private key
aws_access_key_id AWS AKIA… access key ID
aws_secret_access_key AWS secret key
gcp_service_account_key GCP service account JSON key
gcp_api_key GCP API key
azure_connection_string Azure storage connection string
azure_sas_token Azure SAS token
anthropic_api_key Anthropic sk-ant-… key
openai_api_key OpenAI sk-… key
huggingface_token HuggingFace access token
cohere_api_key Cohere API key
stripe_secret_key Stripe sk_live_… / sk_test_… key
github_pat GitHub personal access token (ghp_…)
github_fine_grained_pat GitHub fine-grained PAT
gitlab_pat GitLab PAT (glpat-…)
slack_bot_token / slack_app_token / slack_webhook_url Slack credentials
postgresql_connection_string PostgreSQL DSN
mysql_connection_string MySQL DSN
mongodb_connection_string MongoDB DSN
redis_url / redis_auth_url Redis connection string
dea_number DEA registration number (checksum-validated)
api_key_generic Generic sk-… / AKIA… / ghp_… / glpat-… / xox… patterns

Redact tier entities (threshold 0.55–0.60)

Section titled “Redact tier entities (threshold 0.55–0.60)”
Entity type Threshold Description
ssn 0.55 US Social Security Number
credit_card 0.55 Payment card (Luhn + BIN validated)
itin 0.55 Individual Taxpayer Identification Number
canadian_sin 0.55 Canadian Social Insurance Number
uk_nino 0.55 UK National Insurance Number
passport_us / passport_uk / passport_canadian 0.50 Passport numbers
drivers_license 0.55 Driver’s license
medicare_hic 0.55 Medicare Health Insurance Claim number
iban 0.55 IBAN (mod-97 validated)
aba_routing 0.55 ABA routing number
ein 0.55 Employer Identification Number
npi 0.55 National Provider Identifier (Luhn validated)
medical_record_number 0.55 Medical record number
health_plan_id / ndc_code / hcpcs_code / icd10_code 0.55 Healthcare identifiers
jwt_token / bearer_token 0.60 Auth tokens
stripe_publishable_key 0.60 Stripe publishable key

Log-only tier entities (threshold 0.40–0.45)

Section titled “Log-only tier entities (threshold 0.40–0.45)”
Entity type Threshold Description
email 0.40 Email address
phone_number / phone 0.45 Phone number
swift_bic 0.40 SWIFT/BIC code
crypto_wallet 0.40 Cryptocurrency wallet address
ip_address 0.40 IPv4/IPv6 address
mac_address 0.40 MAC address
network_cidr 0.40 CIDR block

The default threshold for unlisted entity types is 0.50.

These thresholds are configured in the ENTITY_THRESHOLDS constant in the DLP pipeline. Policy rule entity_confidence_min conditions can raise (but not lower) the effective threshold for a specific enforcement action.


Presidio bridge and custom recognizer registration

Section titled “Presidio bridge and custom recognizer registration”

When DLP_NER_BACKEND=default (CPU-based Presidio NER), the pipeline registers custom recognizers for every pattern in the expanded pattern registry. This allows Presidio’s AnalyzerEngine to detect the full set of 70+ entity types alongside its built-in recognizers.

At startup, register_custom_recognizers() iterates over get_all_patterns() (all four DLP modules) and creates a Presidio PatternRecognizer for each pattern definition. The recognizer uses the same regex as the Tier 1 RegexDetector, ensuring consistent detection across both paths.

Presidio ships built-in recognizers for several entity types (credit cards, SSNs, phone numbers) that have lower precision than the Arbitex Tier 1 patterns. At initialization, disable_inferior_builtins() removes these recognizers from the AnalyzerEngine to prevent them from producing low-confidence duplicates of matches that the Tier 1 RegexDetector already handles with checksum validation.

DLP_NER_BACKEND Detector used Notes
default NERDetector (Presidio) CPU-only, no GPU required; uses custom recognizer registry
gliner GLiNERDetector (in-process GLiNER) Requires gliner package; GPU optional but recommended
microservice MicroserviceNERDetector Calls http://ner-gpu:8200; requires NER GPU microservice

Empirical F1 scores and precision/recall metrics are published in the DLP accuracy reference. That page covers:

  • Tier 1 per-pattern F1, precision, and recall scores (71 entity types, 92,944 examples)
  • Tier 3 DeBERTa v4 per-entity F1 scores (35 entity types, 281K examples, 99.48% accuracy)
  • Known limitations (patterns with low F1 in corpus v2.0)
  • Guidance for using entity_confidence_min to tune precision vs recall

For adversarial results, v2-to-v4 improvements, and over-hinting validation, see DeBERTa Tier 3 admin guide.


The DLP pipeline is the detection layer. The Policy Engine is the enforcement layer. They are complementary: the DLP pipeline produces findings; the Policy Engine decides what action to take.

flowchart TD
    Input["Text Input"]
    Input --> T1["Tier 1: Pattern Matching\n70+ regex patterns + checksum validation\n<1ms latency"]
    T1 --> T2["Tier 2: Entity Recognition (NER)\nGPU-accelerated microservice\n60-160ms latency"]
    T2 --> T3["Tier 3: Contextual Validation\nDeBERTa NLI model\n17-90ms latency"]
    T3 --> Combine["Combine all findings"]
    Combine --> PE["Policy Engine\nevaluates findings\nagainst policy chain"]

    style T1 fill:#e8f5e9
    style T2 fill:#fff3e0
    style T3 fill:#fce4ec

DLP findings do not directly block requests. The Policy Engine consumes those findings and determines the enforcement action. A DLP finding includes the entity_type, tier, confidence (0.0–1.0), offset, and length of the detected span. Policy rules reference findings through the entity_types and entity_confidence_min conditions.


The DLP pipeline intercepts every AI request and response as an inline inspection layer — the gateway holds the request until inspection completes before forwarding to the AI provider or returning the response to the client.

Client → [Arbitex Outpost] → [Arbitex Platform] → [AI Provider]
DLP Pipeline (inline)
Policy Engine (inline)

The pipeline runs twice per conversation turn:

Inspection point Text inspected Blocking capable
Request phase User prompt text Yes — can block before sending to provider
Response phase Provider response text Yes — can block/redact before returning to client

Request inspection is synchronous. Response inspection is synchronous for block and redact actions; flag-only rules on responses are evaluated asynchronously to reduce added latency.


When the Policy Engine matches a rule, the Action Dispatcher executes the policy decision. There are four actions:

No intervention. The request is forwarded to the AI provider unchanged. An audit event is created with action: "allow" and the full findings list.

Sensitive spans are replaced with placeholder tokens before the text is forwarded. When multiple overlapping findings are redacted, the longest span is redacted. The AI provider receives the redacted text; if the provider’s response contains the original text, response inspection runs again and redacts before it reaches the client.

The request is dropped. The AI provider is not called. The gateway returns a structured error response to the client:

HTTP/1.1 400 Bad Request
Content-Type: application/json
{
"error": {
"type": "content_policy_violation",
"code": "dlp_block",
"message": "Your request was blocked by a content policy rule.",
"rule_name": "block-credit-card-data",
"request_id": "req_abc123",
"findings_summary": [
{ "entity_type": "credit_card", "count": 1 }
]
}
}

The findings_summary includes entity type and count but NOT the matched text, to avoid leaking sensitive data back to the client.

For response-phase blocking:

HTTP/1.1 502 Bad Gateway
{
"error": {
"type": "response_policy_violation",
"code": "dlp_response_block",
"message": "The AI provider response was blocked by a content policy rule.",
"request_id": "req_abc123"
}
}

The request is allowed to proceed but an alert event is created and sent to configured alert channels. flag rules do not stop processing — the next matching rule’s action still applies. The alert payload includes findings metadata only; entity_text is never included in alert payloads.


Every request that passes through the DLP pipeline creates an audit event, regardless of action taken. Audit events are written asynchronously to the audit_logs PostgreSQL table after the action is dispatched.

The entity_text (the actual sensitive content detected) is intentionally NOT stored in audit events. Audit logs record that sensitive data was detected and where, but do not re-store the sensitive content. This satisfies GDPR and SOC 2 requirements for minimizing sensitive data exposure.

Audit events include a content_hash — an HMAC-SHA256 computed over (request_id + org_id + timestamp + findings_json) using a per-org HMAC key for tamper detection. See Audit Chain Integrity for verification procedures.