DLP detection hierarchy

The Arbitex DLP pipeline does not stop at the first match. Every tier runs against the full text, every match is collected, and only after all detection is complete does the pipeline resolve which action to take. This guide explains exactly how that process works — from initial regex scan through policy engine evaluation — so you can predict what will happen when a request contains multiple entity types, overlapping spans, or mixed sensitivity levels.

If you are looking for how to configure detection tiers, see DLP Pipeline Configuration. If you are looking for accuracy measurement methodology, see DLP Accuracy Validation. This guide covers the detection mechanics themselves.

---

Detection is exhaustive

The pipeline processes text through four detection tiers in sequence. The key design principle: all tiers run against the full text, and all matches are collected before any action is taken.

Consider a prompt that contains a Social Security Number, a credit card number, and a date of birth. The pipeline does not detect the SSN, decide to block, and skip the remaining scan. Instead:

1. Tier 1 (Regex) scans the full text — finds the SSN and the credit card number
2. Tier 2 (NER) scans the full text — finds the SSN (contextually), the credit card, and the date of birth
3. Tier 3 (DeBERTa) validates all candidate matches — confirms or demotes each one
4. Tier 4 (CredInt) scans for compromised credentials concurrently

All surviving matches from all tiers are collected into a single result set. The pipeline then deduplicates overlapping spans, applies confidence filtering, resolves the final action from the highest-priority match, and records every detected entity in the audit trail.

A BLOCK action on the SSN does not suppress detection of the credit card or date of birth. All three entities appear in the audit trail regardless of which action is taken.

Platform BLOCK short-circuit

The platform implementation includes a performance optimization: if a detector produces a match whose configured action is BLOCK, the pipeline skips remaining detectors (since the outcome — block the request — is already determined). Even in this case, all matches from detectors that did run are preserved in the audit trail. The outpost implementation only short-circuits after Tier 1 (regex) produces a BLOCK; Tiers 2–4 always run if Tier 1 did not block.

---

The four detection tiers

Each tier uses a different detection technology with different strengths. Running all tiers provides defense in depth — what one tier misses, another can catch.

Tier 1: Regex pattern matching

Property	Value
Technology	Compiled regular expressions with structural validators
Pattern count	76 built-in patterns across 6 categories
Confidence	Always 1.0 for validated matches
Latency	< 1 ms per request
Execution	In-process, synchronous

Tier 1 is the foundation. It scans for structured data formats — credit card numbers, SSNs, API keys, IBANs, JWTs, PEM-encoded keys — using pattern matching with checksum and structural validation. A regex match against a credit card pattern includes Luhn validation; a match against an IBAN pattern includes the modulo-97 check. This structural validation is what gives Tier 1 its high precision.

The 76 patterns are organized into categories:

Category	Count	Examples
Base PII	4	Credit card, SSN, email, phone
Secrets	39	AWS keys (AKIA...), GitHub PATs, JWTs, PEM headers, database URIs, bearer tokens
PII	12	Passport numbers, driver licenses, ITIN, EIN, Indian PAN/Aadhaar
Financial	8	IBAN, SWIFT/BIC, routing numbers, crypto wallet addresses
Medical	7	DEA numbers, NPI, NHS numbers, Canadian health cards
Infrastructure	6	IP addresses, connection strings, SAS tokens

Tier 1 is deterministic: given the same input text and the same pattern set, it produces the same matches every time. There is no model inference, no probabilistic scoring. A pattern either matches or it does not.

Tier 2: Named Entity Recognition (NER)

Property	Value
Technology	Presidio AnalyzerEngine (platform) / spaCy en_core_web_sm (outpost)
Default entities	PERSON, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, US_SSN, IP_ADDRESS, IBAN_CODE
Confidence	Varies by entity type (0.75–0.85 typical)
Latency	60–160 ms per request
Execution	In-process or GPU microservice

Tier 2 uses machine learning to identify entities that regex cannot reliably catch. NER recognizes “my social security number is 078051120” from surrounding context even when the number lacks the standard XXX-XX-XXXX format. It catches names, organizations, and location references that are relevant to PII classification but have no fixed pattern.

The platform supports three NER backends, selected by the DLP_NER_BACKEND environment variable:

Backend	Technology	Use case
default	Presidio with custom recognizers	Standard deployment — bridges all 72 platform regex patterns through Presidio’s context-scoring layer
gliner	GLiNER zero-shot NER (urchade/gliner_medium-v2.1)	Enhanced recall for novel entity types without retraining
microservice	GPU sidecar service	High-throughput deployments requiring dedicated inference hardware

NER confidence scores are ordinal rather than probabilistic. A confidence of 0.85 from Tier 2 means “high certainty from the NER model” but does not correspond to a calibrated 85% precision rate. Use the confidence calibration analysis to understand what Tier 2 scores mean in practice.

Tier 3: DeBERTa NLI validation

Property	Value
Technology	DeBERTa v3 (microsoft/deberta-v3-base) natural language inference
Role	Second-pass validator — confirms or demotes Tier 1/2 candidates
Confidence	Well-calibrated probabilities (tracks actual precision closely)
Latency	~17–90 ms per request (ONNX Runtime)
Execution	GPU microservice (platform) / in-process ONNX (outpost)

Tier 3 is not a standalone detector. It runs after Tiers 1 and 2 and evaluates each candidate match in context. The model performs natural language inference against sensitivity hypotheses: given the surrounding text, is this span actually sensitive data, or is it a benign value that happens to match a pattern?

This is what distinguishes “the test SSN 078-05-1120 from IRS Publication 1346” (not sensitive) from “patient SSN: 078-05-1120” (sensitive). Tier 1 fires on both — the format matches. Tier 3 uses the surrounding context to demote the test value below the confidence threshold while confirming the real one.

Platform behavior: DeBERTa runs as a validator injected into the pipeline after all detectors complete. It receives the full list of raw matches and can adjust confidence scores. Matches demoted below the confidence threshold are filtered out before action resolution.

Outpost behavior: DeBERTa runs as a binary classifier (pii / clean) on 450-character text chunks via ONNX Runtime. The internal confidence threshold is 0.7 — stricter than the pipeline default of 0.5.

Tier 3 only sees candidates surfaced by Tiers 1 and 2. If neither tier detects an entity, Tier 3 never evaluates it. This is by design — Tier 3 improves precision (reducing false positives), not recall.

Tier 4: Credential Intelligence (CredInt)

Property	Value
Technology	Bloom filter (industry-leading compromised credential dataset) with optional k-anonymity verification
Detection target	Known-compromised credentials
Confidence	1.0 - bloom_filter_fpr (effectively ~0.9999)
Action	Always BLOCK
Execution	Concurrent with Tiers 1–3

Tier 4 operates differently from the other tiers. It does not look for patterns or entity types — it checks whether credential-like tokens (email:password pairs, API keys, bearer tokens) appear in a corpus of known-compromised credentials from public breach datasets.

The bloom filter uses double hashing (h_i(x) = (h1(x) + i × h2(x)) % m) via SHA-256. Token extraction pulls colon-separated credential pairs, key-value credential patterns, and API key/token formats from the text.

Key CredInt behaviors:

• Privacy by design: credential token values are never included in the entity output. The detection record contains the entity type and span, but text is always empty.
• Action is always BLOCK: a compromised credential in transit is a security incident, not a policy decision.
• Concurrent execution: CredInt runs in parallel with the synchronous tiers, not sequentially after them.
• Refresh mechanism: the bloom filter can be updated from CDN with downgrade protection (new filter must be same size or larger).

For full CredInt architecture, see Credential Intelligence.

---

Multi-match detection: worked example

To make the exhaustive detection behavior concrete, here is what happens when a prompt contains three different entity types.

Input prompt:

Please update the records for patient John Smith.
SSN: 078-05-1120, credit card ending 4532-0151-1283-0366,
date of birth 1985-03-15. API key: sk-proj-abc123def456ghi789.

Tier 1 (Regex) results:

Entity type	Span	Confidence	Validator
SSN	078-05-1120	1.0	Format valid (XXX-XX-XXXX)
CREDIT_CARD	4532-0151-1283-0366	1.0	Luhn-valid
API_KEY	sk-proj-abc123def456ghi789	1.0	Prefix match (sk-proj-)

Tier 2 (NER) results:

Entity type	Span	Confidence
PERSON	John Smith	0.85
SSN	078-05-1120	0.82
CREDIT_CARD	4532-0151-1283-0366	0.80
DATE	1985-03-15	0.80

Raw match set (before deduplication): 7 matches across 5 entity types.

Tier 3 (DeBERTa) validation: DeBERTa evaluates each candidate in context. The SSN appears in a medical record context — confirmed as sensitive (confidence maintained). The credit card appears with explicit labeling — confirmed. The API key has the sk-proj- prefix in a “please update” context — confirmed. The person name appears in a patient context — confirmed as PII.

After deduplication: The SSN span 078-05-1120 was detected by both Tier 1 (confidence 1.0) and Tier 2 (confidence 0.82). Since both detections cover the same span and the same entity type, deduplication keeps the higher-confidence match (Tier 1, confidence 1.0).

Final match set: 5 entities — SSN, CREDIT_CARD, PERSON, DATE, API_KEY.

All five entities are recorded in the audit trail, regardless of which action the pipeline takes.

---

Deduplication: resolving overlapping matches

When multiple tiers detect the same span of text, the pipeline must decide which matches to keep. The deduplication rules differ between platform and outpost implementations, but both follow the same core principle: different entity types on overlapping spans are preserved; same entity type on the same span keeps the highest-confidence match.

Same entity type, overlapping span

When Tier 1 detects an SSN at positions 50–61 with confidence 1.0, and Tier 2 detects an SSN at positions 50–61 with confidence 0.82, these are deduplicated. The Tier 1 match wins because it has higher confidence.

Different entity types, overlapping span

When Tier 1 detects a PHONE at positions 100–114, and Tier 2 detects an SSN at positions 100–109 (overlapping but not identical span), both matches are preserved. The pipeline keeps both because they represent different entity types — the same text legitimately could be both a phone number fragment and an SSN.

Deduplication sort order

Matches are sorted by:

1. Start position (ascending)
2. Confidence (descending)
3. Tier priority (higher tiers preferred for tie-breaking)

This ensures deterministic results regardless of the order in which detectors return matches.

---

Confidence threshold filtering

After deduplication, matches below the confidence threshold are discarded. The default threshold is 0.5 — any match with confidence below 0.5 is filtered out before action resolution.

Component	Default threshold	Notes
Pipeline global	0.5	Applies to all matches after DeBERTa validation
DeBERTa internal (outpost)	0.7	Applied within the DeBERTa scanner before results enter the pipeline
GLiNER internal	0.5	Matches the pipeline default
Tier 1 (Regex)	N/A	Confidence is always 1.0 for validated matches — never filtered
Tier 4 (CredInt)	N/A	Confidence ≈ 1.0 — never filtered

The threshold is configurable per deployment. Lowering the threshold increases recall (catches more edge cases) at the cost of precision (more false positives). Raising it does the opposite. See DLP Pipeline Configuration for tuning guidance.

---

Action hierarchy and resolution

Every entity type has a configured action that determines what happens when that entity is detected. The four actions, in priority order:

LOG_ONLY (0) < PROMPT (1) < REDACT (2) < BLOCK (3)

Action	Priority	Behavior
LOG_ONLY	0	Record the detection in the audit trail. Allow the request to proceed unchanged.
PROMPT	1	Challenge the user with a governance prompt before allowing the request. Requires channel=interactive.
REDACT	2	Replace the detected span with a redaction marker and allow the request to proceed with the sensitive data removed.
BLOCK	3	Reject the request entirely. No content reaches the model provider.

Highest-priority action wins

After all matches are collected, deduplicated, and confidence-filtered, the pipeline resolves the final action by selecting the highest-priority action across all surviving matches.

If the policy configuration is:

• SSN → BLOCK
• CREDIT_CARD → REDACT
• PERSON → LOG_ONLY

And the prompt contains all three entity types, the final action is BLOCK (priority 3, from the SSN match). The credit card and person name are still recorded in the audit trail, but the request is blocked because at least one entity type triggered BLOCK.

Redaction markers

When the resolved action is REDACT, detected spans are replaced with markers:

• Platform: [REDACTED] (flat marker, no entity type in the replacement)
• Outpost: [REDACTED:{entity_type}] (typed marker, e.g., [REDACTED:credit_card])

The typed marker format in the outpost allows downstream systems to understand what was redacted without accessing the original text.

Action override by policy rules

The action configured per entity type is the default. Policy rules can override the action for specific contexts — for example, allowing LOG_ONLY for SSNs in a specific application while keeping BLOCK as the global default. See Policy engine interaction below.

---

Detection flow diagram

The complete detection pipeline, from text input to final action:

┌─────────────────────────────────────────────────────────┐
│                    INPUT TEXT                            │
└────────────────────────┬────────────────────────────────┘
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
   ┌─────────────┐ ┌──────────┐ ┌────────────┐
   │   Tier 1    │ │  Tier 2  │ │   Tier 4   │
   │   Regex     │ │   NER    │ │  CredInt   │
   │ (76 patterns│ │(Presidio/│ │(bloom filter│
   │  + checks)  │ │  spaCy)  │ │ concurrent)│
   └──────┬──────┘ └────┬─────┘ └─────┬──────┘
          │              │             │
          ▼              ▼             │
   ┌─────────────────────────────┐     │
   │    Raw match collection     │     │
   │  (all Tier 1+2 matches)     │     │
   └──────────────┬──────────────┘     │
                  ▼                    │
   ┌─────────────────────────────┐     │
   │       Tier 3: DeBERTa       │     │
   │   NLI validation pass       │     │
   │  (confirms or demotes each  │     │
   │   candidate in context)     │     │
   └──────────────┬──────────────┘     │
                  ▼                    │
   ┌─────────────────────────────┐     │
   │  Confidence threshold filter│     │
   │  (default ≥ 0.5)           │     │
   └──────────────┬──────────────┘     │
                  ▼                    ▼
   ┌─────────────────────────────────────────┐
   │         Merge all surviving matches      │
   │    (Tiers 1–3 filtered + Tier 4 hits)    │
   └──────────────────┬──────────────────────┘
                      ▼
   ┌─────────────────────────────────────────┐
   │            Deduplication                 │
   │  Same type + overlap → keep highest     │
   │  Different type + overlap → keep both   │
   └──────────────────┬──────────────────────┘
                      ▼
   ┌─────────────────────────────────────────┐
   │         Action resolution               │
   │  highest_action = max(all match actions)│
   │  LOG_ONLY < PROMPT < REDACT < BLOCK     │
   └──────────────────┬──────────────────────┘
                      ▼
   ┌─────────────────────────────────────────┐
   │     All detected entities recorded      │
   │     in audit trail (regardless of       │
   │     which action was taken)             │
   └──────────────────┬──────────────────────┘
                      ▼
   ┌─────────────────────────────────────────┐
   │         PolicyContext built             │
   │  detected_entities → policy chain eval  │
   └─────────────────────────────────────────┘

---

Policy engine interaction

DLP scanning completes first. The pipeline produces a list of detected entities with their types, confidence scores, spans, and configured actions. This list is then packaged into a PolicyContext and fed into the policy chain for evaluation.

PolicyContext and detected_entities

The policy engine receives the full DLP result set through the detected_entities field:

PolicyContext {
  user_id
  tenant_id
  provider
  model
  prompt_text
  detected_entities: [          ← ALL DLP matches
    { entity_type, confidence, start, end, tier, action },
    { entity_type, confidence, start, end, tier, action },
    ...
  ]
  user_groups
  user_risk_score
  direction: "input" | "output"
  channel: "interactive" | "api"
  content_categories
}

Policy rule matching on entities

Policy rules can match on detected entity types with optional confidence minimums:

{
  "conditions": {
    "entity_types": ["ssn", "credit_card"],
    "entity_confidence_min": 0.8
  },
  "action": "BLOCK"
}

This rule fires if any detected entity is of type ssn or credit_card with confidence ≥ 0.8. The entity_confidence_min condition prevents low-confidence matches from triggering high-severity policy actions.

Policy evaluation modes

The policy engine supports two evaluation strategies (following the Palo Alto / XACML convention):

Mode	Behavior
first_applicable	Rules are evaluated in order. The first rule whose conditions match determines the action. Remaining rules are not evaluated.
deny_overrides	All rules are evaluated. If any rule produces a deny action (BLOCK or CANCEL), that overrides any allow actions from other rules.

Terminal vs. non-terminal actions

Within the policy engine, actions have different termination behavior:

Action	Terminal?	Behavior
ALLOW	Yes	Request proceeds, evaluation stops
PROMPT	Yes	Governance challenge issued, evaluation stops
REDACT	No	Redaction applied, evaluation continues with remaining rules
BLOCK	Yes	Request denied, evaluation stops
CANCEL	Yes	Request denied (silent), evaluation stops

REDACT is non-terminal — after redaction is applied, the policy engine continues evaluating subsequent rules. This means a request can be both redacted (sensitive spans removed) and then evaluated by additional rules that check for other conditions.

Action severity in policy engine

The policy engine uses its own action severity scale for deny_overrides resolution:

ALLOW (0) < PROMPT (1) < ROUTE_TO (2) < REDACT (3) < CANCEL (4) < BLOCK (5)

This is broader than the DLP action hierarchy because the policy engine handles routing and cancellation in addition to DLP actions.

---

Per-org customization

Organizations can customize DLP detection behavior through two mechanisms:

Suppression rules

Exclude specific built-in patterns by name. If a built-in regex pattern produces false positives for a specific organization’s data (for example, internal product codes that match credit card patterns), a suppression rule removes those matches after the main pipeline scan.

Custom pattern rules

Add organization-specific regex patterns that run alongside the built-in patterns. Custom patterns always have confidence 1.0, default to entity type org_custom_pattern, and default action LOG_ONLY. Organizations can override both the entity type and action per custom rule.

Custom rules are cached with a 60-second TTL per organization.

---

Entity type taxonomy

The platform organizes entity types into eight canonical categories:

Category	Key entity types
Direct identifiers	name, address, telephone, email, username
Employment & financial	employment info, account balances, contract numbers
Digital & location	IP address, geolocation, website tracking
Government IDs	SSN, driver license, passport, ITIN, NPI, DEA number
Financial instruments	bank account, credit card, IBAN, SWIFT/BIC, crypto wallet
Protected health & sensitive	health info, biometric, genetic data (GDPR Art. 9 categories)
Credentials	API key, private key, bearer token, connection string
MNPI	earnings announcements, M&A, insider information, material contracts

The normalize_entity_type() function resolves aliases to canonical names, ensuring consistent entity type references across detection tiers and policy rules.

---

Key behaviors summary

Behavior	Detail
Detection scope	Exhaustive — all tiers scan the full text
Match collection	All matches from all tiers are collected before action resolution
BLOCK short-circuit	Platform: skips remaining detectors (performance optimization). Outpost: only after Tier 1 regex BLOCK. All matches from executed tiers are preserved.
Deduplication	Same type + overlap → highest confidence wins. Different types + overlap → both kept.
Confidence threshold	0.5 default. Matches below threshold are discarded after DeBERTa validation.
Action resolution	Highest-priority action across all surviving matches determines the outcome
Audit trail	All detected entities are recorded regardless of the action taken
Policy interaction	DLP completes first → PolicyContext.detected_entities → policy chain evaluation

---

Related documentation

• DLP Pipeline Configuration — enable/disable tiers, add custom rules, tune thresholds
• DLP Accuracy Validation — measure detection accuracy with golden corpus and CI gates
• DLP Event Monitoring — monitor DLP detections via the audit log
• DLP Redaction Visibility — how redaction is presented to users and admins
• DLP Regulatory Patterns — regulatory compliance pattern coverage
• Credential Intelligence — Tier 4 architecture, corpus, and lookup protocol
• DLP Pipeline Architecture — technical reference for pipeline internals