DLP Pipeline Architecture

The Arbitex data loss prevention (DLP) pipeline inspects every prompt and every model response before it reaches its destination. It runs in two directions — input (before the provider call) and output (after it) — and applies a 5-tier cascade of progressively more powerful detection methods. This document describes the full pipeline architecture, the detection capabilities of each tier, redaction mechanics, and how organisations extend or restrict the default behavior.

ScanEngine orchestration

The DLP pipeline is orchestrated by ScanEngine (arbitex_core.dlp.scan_engine), the unified extraction and scanning library in arbitex-core. The legacy DLPPipeline class is deprecated — all channels now use ScanEngine for both content extraction and DLP scanning. See ScanEngine Architecture for the full design.

ScanEngine Phase 3 capabilities:

• Early termination — A BLOCK action at any tier short-circuits the remaining pipeline. No downstream tiers execute.
• Async detector support — CredInt (Tier 4) runs as a concurrent asyncio.create_task alongside the Tier 1→2→3 chain, avoiding serial latency.
• Per-tier OTel metrics — Each tier emits OpenTelemetry metrics for decision counts, latency, and error rates.
• Entity normalization — Detector output is normalized to a consistent DLPMatch schema before deduplication.

---

Pipeline Overview

The pipeline is a cascade: each tier only runs if the previous tier did not produce a terminal (BLOCK) action. Tier 0 runs on input only and short-circuits the entire pipeline on high-confidence prompt injection. The CredInt tier runs concurrently with the L1→L2→L3 chain rather than serially, to avoid adding its latency to the critical path.

flowchart TD
    A([Text Input\nprompt or response]) --> SE[ScanEngine\norchestrator]

    SE --> T0

    subgraph T0["Tier 0 — TF-IDF Prompt Injection Pre-filter (input only)"]
        T0_CLASS[Tier0Classifier\nTF-IDF + LogisticRegression\ntier0_model.joblib]
        T0_CLASS --> T0_CONF{confidence\n≥ threshold?}
        T0_CONF -->|Yes — block| T0_BLOCK([BLOCK — early termination\nskip all remaining tiers])
        T0_CONF -->|No — pass| T0_PASS[Continue to Tier 1]
        T0_NOTE[Skipped for output direction\nand when model is absent]
    end

    T0_PASS --> T1

    subgraph T1["Tier 1 — Regex Pattern Matching"]
        R1[RegexDetector\ndlp_patterns.py + dlp_patterns_pii.py\n+ dlp_patterns_financial.py\n+ dlp_patterns_medical.py]
        R1 --> OA{Match\naction_tier?}
        OA -->|block| EARLY_BLOCK([BLOCK — early termination\nskip Tiers 2–3])
        OA -->|redact / log_only| PASS1[Continue to Tier 2]
        OA -->|no match| PASS1
    end

    PASS1 --> T2

    subgraph T2["Tier 2 — NER Entity Extraction"]
        N1[MicroserviceNERDetector\n→ GPU NER microservice\nPOST /detect]
        N2[Presidio bridge\ndlp_presidio_bridge.py]
        N3[MNPIGLiNERDetector\ngliner_detector.py]
        N1 & N2 & N3 --> MERGE2[Merge + deduplicate entities]
    end

    MERGE2 --> T3

    subgraph T3["Tier 3 — DeBERTa Contextual Validation\n(v4 production · v6 planned)"]
        D1[DeBERTaValidatorClient\ndlp_deberta.py\nPOST /validate]
        D1 --> CONF{Confidence\nthreshold?}
        CONF -->|P > 0.70| HARD_BLOCK[hard_block]
        CONF -->|0.35–0.70| AMBIG[Sensitivity-dependent routing]
        CONF -->|P < 0.35| PASS3[pass]
    end

    HARD_BLOCK --> ACTION
    AMBIG --> ACTION
    PASS3 --> ACTION

    subgraph T4["Tier 4 — CredInt Credential Intelligence\n(runs concurrently via asyncio.create_task)"]
        C1[run_credint_check\ncredential_extractor → CredentialIntelligenceClient]
        C1 --> CB{frequency\nbucket}
        CB -->|critical / high| ROUTE_C[Route by dlp_sensitivity]
        CB -->|medium / low| FLAG[pass + audit_flag]
        CB -->|no hit| CLEAN[pass]
    end

    ROUTE_C & FLAG & CLEAN --> CREDINT_RESULT[CredInt audit fields\nmerged into extra_metadata]

    subgraph ACTION["Decision & Action"]
        OrgLayer[OrgDLPLayer\nCustom patterns + suppression rules]
        BundleFilter[filter_scan_result_by_entity_types\nCompliance bundle enforcement]
        FinalAction{Final action}
    end

    CREDINT_RESULT --> ACTION

    FinalAction -->|LOG_ONLY| AuditLog([Audit log entry\n+ OTel metrics])
    FinalAction -->|REDACT| Redact([Redact matched spans\npass sanitised text])
    FinalAction -->|BLOCK| Block([Block request / response])

    Redact & AuditLog & Block --> WebhookFire[fire_dlp_trigger\ndlp_trigger webhook\non BLOCK or REDACT]

Pipeline position in the request lifecycle

The DLP pipeline is invoked twice per request:

1. Input scan — during Stage 2 of the intake pipeline (intake_pipeline.py). Tier 0 runs first; if it blocks, the remaining tiers are skipped. Otherwise, the L1→L2→L3 chain runs in parallel with quota/rate checks via asyncio.gather. A BLOCK action at this stage returns 403 to the caller and aborts the provider call. CredInt runs as a non-blocking asyncio.create_task alongside the L1→L2→L3 scan chain.

2. Output scan — after the provider returns. The OutputScanner class accumulates streamed response chunks and triggers incremental scans at configurable character intervals (default: 500 characters). A BLOCK on output suppresses the response and fires an output_blocked SSE event. The feature is gated by the DLP_OUTPUT_SCANNING_ENABLED environment variable (default: true).

---

Tier 0: TF-IDF Prompt Injection Pre-filter

Module: backend/app/services/tier0_classifier.py (platform), outpost/tier0_classifier.py (outpost)

Tier 0 is a lightweight TF-IDF + logistic regression classifier that runs on input only before the full detection pipeline. Its purpose is to catch obvious prompt injection attempts and short-circuit processing, eliminating the latency cost of subsequent tiers for clearly malicious inputs.

How it works

The classifier is a serialized scikit-learn Pipeline (TF-IDF vectorizer + logistic regression) loaded from tier0_model.joblib. It produces a binary classification (injection vs. clean) with a confidence score:

• Class 1 = prompt injection, Class 0 = clean
• should_block = is_injection AND confidence ≥ threshold
• Default confidence threshold: 0.95 (configurable via tier0_confidence_threshold system config)

When Tier 0 blocks, it produces a single DLPMatch with entity_type: "prompt_injection" and detector_name: "tier0", and returns a BLOCK action immediately. Tiers 1–4 do not run.

Fail behavior

Scenario	Platform	Outpost
Model file missing	Fail-open (tier disabled)	Fail-open (tier disabled)
Classification error	Fail-closed after 1 retry (blocks)	Fail-open (passes)
Output direction	Skipped (input only)	Skipped (input only)

Configuration

Setting	Type	Default	Description
tier0_prompt_injection_enabled	system config	true	Master toggle
tier0_confidence_threshold	system config	0.95	Minimum confidence to block
TIER0_MODEL_PATH	env var	backend/data/tier0_model.joblib (platform) / outpost/data/tier0_model.joblib (outpost)	Model file path override

OTel metrics

• tier0_decisions_total — counter with labels: decision (block/pass), confidence_bucket (high ≥ 0.95, medium ≥ 0.70, low < 0.70)
• tier0_classify_fail_closed_total — counter for fail-closed events (platform only)

For configuration guidance, see Tier 0 Pre-filter Guide.

---

Tier 1: Pattern Matching

Module: arbitex_core.dlp.patterns (canonical definitions). The platform modules backend/app/core/dlp_patterns.py, dlp_patterns_pii.py, dlp_patterns_financial.py, and dlp_patterns_medical.py are re-export shims that import from the core library.

Tier 1 is the fastest detection layer. All patterns are compiled re.Pattern objects; scanning is fully synchronous and runs inside the gateway process with no external calls.

Pattern registry

All patterns are typed using the SecretPattern TypedDict defined in dlp_patterns.py:

class SecretPattern(TypedDict):
    name: str
    regex: re.Pattern[str]
    entity_type: str
    confidence_threshold: float   # 0.0–1.0; secrets use 0.9+
    action_tier: str              # "block" or "redact"
    category: str                 # "secret", "pii", "financial", "medical", "infrastructure"
    validator: NotRequired[Callable[[str], bool]]  # optional checksum validator

The ALL_PATTERNS list (accessed via get_all_patterns()) is the unified registry that combines all category modules and is loaded by both the RegexDetector and the Presidio bridge.

Secret patterns (dlp_patterns.py)

~39 patterns across six categories:

Category	Examples
Cloud keys	aws_access_key_id (AKIA…), gcp_api_key (AIza…), azure_connection_string, azure_sas_token, aws_secret_access_key, gcp_service_account_key
AI/ML tokens	anthropic_api_key (sk-ant-…), openai_api_key (sk-…), huggingface_token (hf_…), cohere_api_key
Payment/SaaS	stripe_secret_key (sk_live_…), stripe_publishable_key, twilio_auth_token, sendgrid_api_key (SG.…), mailgun_api_key, square_access_token
Communication	discord_webhook_url, slack_webhook_url, slack_bot_token (xoxb-…), slack_app_token (xapp-…), telegram_bot_token
Infrastructure	pem_private_key, ssh_private_key, postgresql_connection_string, mysql_connection_string, mongodb_connection_string, redis_url
Auth tokens	jwt_token, bearer_token, github_pat (ghp_…), github_fine_grained_pat, gitlab_pat (glpat-…), npm_token, pypi_token, docker_hub_token, github_oauth_token, github_app_token, bitbucket_app_password, nuget_api_key

All secret patterns use action_tier: "block" except JWT tokens and Bearer tokens, which use "redact".

PII patterns (dlp_patterns_pii.py)

12 patterns with structural validators to suppress false positives:

Pattern name	Entity type	Validator
itin	itin	validate_itin() — checks 9xx prefix and group number ranges
canadian_sin	canadian_sin	validate_sin_luhn() — Luhn mod-10 checksum
uk_nino	uk_nino	Negative lookahead for invalid prefixes (BG, GB, NK, KN, TN, NT, ZZ) and first-letter codes (D, F, I, Q, U, V)
us_passport, uk_passport, canadian_passport	passport_number	Structure-only
dl_california, dl_texas, dl_florida, dl_new_york, dl_illinois	drivers_license	Structure-only
medicare_hic	medicare_hic	Structure-only

All PII patterns use action_tier: "redact".

Financial patterns (dlp_patterns_financial.py)

Banking, tax, and cryptocurrency patterns:

Category	Patterns
Banking	iban (with validate_iban_mod97() checksum), swift_bic, aba_routing_number
Tax/government	ein (Employer Identification Number)
Cryptocurrency	Bitcoin P2PKH (bitcoin_address), Bitcoin P2SH, Bitcoin Bech32, Ethereum (ethereum_address)

Medical and infrastructure patterns (dlp_patterns_medical.py)

Category	Patterns
Healthcare identifiers	dea_number (with checksum validator), npi_number, mrn, medicare_mbi, ndc_code, hcpcs_code, icd10_code
Network/infrastructure	IPv6 addresses, MAC addresses, CIDR notation, MSSQL connection strings

Compiled Rust scanner backend

The DLP_SCANNER_BACKEND environment variable selects the Tier 1 scanning engine:

Backend	Value	Description
Python (default)	python	Pure-Python re module scanning. No additional dependencies.
Rust (compiled)	rust	Compiled Rust scanner via CompiledScannerCache from arbitex_core.dlp.compiled_scanner. Falls back to Python if the Rust extension (_dlp_scanner_rs) is not installed.

When DLP_SCANNER_BACKEND=rust, the RegexDetector (platform) and RegexScanner (Outpost) initialize a CompiledScannerCache that compiles all regex patterns into a Rust-native multi-pattern automaton at startup. The Rust scanner accepts patterns with inline flag notation ((?i), (?m), (?s), (?x)) — Python re.Pattern objects are converted to Rust-compatible strings by _regex_to_rust_str(), stripping the default re.UNICODE flag and translating remaining flags.

The scan flow under the Rust backend:

1. _build_rust_patterns() converts base + expanded patterns to Rust-compatible dicts, applying org overlay rules (suppress_default / custom_pattern).
2. _scan_rust() delegates multi-pattern matching to the compiled Rust engine.
3. Python-side post-processing applies checksum validators (Luhn for credit cards, IBAN mod-97, ITIN group ranges) and span deduplication — the Rust scanner handles pattern matching only.

Fallback: If DLP_SCANNER_BACKEND=rust is set but the Rust extension is not installed, the detector logs a warning (dlp_rust_scanner_fallback) and falls back to the Python backend transparently. The Python scanning path is completely unchanged — the Rust backend is a drop-in performance optimization with zero functional differences.

Outpost parity: The Outpost implements the same Rust scanner integration. Both the platform RegexDetector and Outpost RegexScanner use CompiledScannerCache from arbitex_core.dlp.compiled_scanner, dispatch to _scan_rust() or _scan_python() based on the feature flag, and apply identical Python-side post-validation. Parity is verified by dedicated test suites on both the platform and Outpost, including extensive parity checks.

NER gate (pack-aware NER gating)

Not all DLP workloads require Tier 2 NER detection. The NER gate (dlp_ner_gate.py) determines whether NER should run for a given request based on three factors:

1. DLP_NER_FORCE override: A system_config key that globally or per-org overrides NER behavior.
2. Compliance pack flags: Each compliance pack declares an ner_required boolean. NER runs only when at least one of the org’s enabled packs requires it.
3. Keyword pre-filter: A regex scan for ~57 NER-trigger keywords (person names, medical terms, dates, locations) that can skip NER on clearly non-PII text like code snippets.

DLP_NER_FORCE values:

Value	Behavior
auto (default)	NER runs only when the org’s enabled compliance packs require it AND trigger keywords are present
always	NER always runs regardless of pack configuration
never	NER is never invoked

Per-org override: set dlp_ner_force:{org_id} in system_config to override the global default for a specific org.

Pack NER requirements (from seed definitions):

Pack	ner_required	Rationale
HIPAA	true	Person names and medical terms
GDPR	true	Person names and locations
CCPA	true	Person names and locations
PCI-DSS	false	Structural patterns only (card numbers, IBANs)
GLBA	false	Structural patterns only
SOX	false	Structural patterns only
BSA-AML	false	Structural patterns only
SEC Reg FD	false	Structural patterns and topic entities only

Resolution is cached with a 60-second TTL (_get_ner_force_config(), _get_org_enabled_frameworks()) to avoid per-request database lookups.

DeBERTa confidence threshold skip

Tier 3 DeBERTa validation skips high-confidence matches from non-NER detectors (confidence ≥ 0.90) where the marginal value of contextual validation is low. NER-originated matches are always validated regardless of confidence, because NER inherently produces lower-confidence results that benefit from contextual confirmation.

Match source	Confidence	DeBERTa behavior
Regex (Tier 1)	≥ 0.90	Skipped — high structural confidence
Regex (Tier 1)	< 0.90	Validated
NER (Tier 2)	Any	Always validated
Validator-confirmed	Any	Skipped (already confirmed)

Skip metrics are logged per validation call: total, skipped, skip_pct, to_validate.

Performance

Tier 1 completes in the sub-millisecond range for typical prompt-length text (1–2 KB). At 1 KB payload size, the outpost benchmark records median (p50) scan times well under 1 ms. Early termination on a BLOCK-action match prevents Tier 2 and Tier 3 from running, keeping the fast path fast.

The optional validator callable is invoked after a regex match and can reject structurally valid but semantically invalid matches (e.g. Luhn-failing card numbers, ITIN with invalid group ranges).

---

Tier 2: NER Entity Extraction

Tier 2 discovers entities that Tier 1 misses because they lack a predictable structural format — person names, organisation names, locations, and natural language expressions of financial amounts.

Three detectors operate at this tier.

MicroserviceNERDetector (dlp_microservice.py)

The primary NER detector. Calls POST /detect on the GPU-backed NER microservice over httpx:

payload = {
    "text": text,
    "labels": [...],  # optional entity type filter
    "threshold": 0.5, # optional confidence cutoff
}
# Response:
{
    "entities": [
        {"text": "John Doe", "label": "person", "start": 0, "end": 8, "score": 0.95},
    ],
    "model": "...", "device": "...", "processing_time_ms": 12.3
}

Responses are parsed into DLPMatch objects by MicroserviceNERDetector._parse_response(). Entity labels are normalised to lowercase with spaces replaced by underscores (e.g. "PERSON" → "person").

Default endpoint: http://ner-gpu:8200 (Docker Compose service name). Override with DLP_NER_MICROSERVICE_URL.

Circuit breaker: After 3 consecutive failures (_CIRCUIT_BREAKER_THRESHOLD), the detector enters an open state and does not make HTTP calls for 60 seconds (_CIRCUIT_BREAKER_RESET_SECONDS). After the reset window expires, one probe request is allowed (half-open state). A successful probe closes the circuit; another failure re-opens it with a refreshed timer.

Fail mode: Controlled by DLP_INFERENCE_FAIL_MODE (default: "closed"). In closed mode, a microservice error or open circuit raises DLPInferenceUnavailableError and blocks the request rather than passing it through unscanned. In open (legacy) mode, errors return an empty match list.

Presidio bridge (dlp_presidio_bridge.py)

The register_custom_recognizers() function converts every entry in ALL_PATTERNS into a Presidio PatternRecognizer, wrapping the compiled re.Pattern in a Pattern object scored at the pattern’s confidence_threshold. This makes the full regex pattern library available to the Presidio AnalyzerEngine as named recognizers:

recognizer = PatternRecognizer(
    supported_entity=pattern_def["entity_type"].upper(),
    name=f"arbitex_{pattern_def['name']}_recognizer",
    patterns=[presidio_pattern],
    supported_language="en",
)

Presidio’s context-scoring layer can then boost confidence for patterns that appear near contextually relevant keywords (e.g. a 16-digit sequence near the word “card”).

GLiNER MNPI detector (gliner_detector.py)

MNPIGLiNERDetector uses zero-shot entity recognition for Material Non-Public Information. It calls GLiNER.from_pretrained() with GLINER_MODEL_NAME (default: urchade/gliner_multi_pii-v1) and predicts against six canonical MNPI entity labels:

MNPI_ENTITY_LABELS = [
    "earnings announcement",
    "merger acquisition",
    "insider information",
    "material contract",
    "regulatory action",
    "executive change",
]

The model is loaded lazily on first detect() call and cached as a thread-safe module-level singleton. If CUDA is available, the model is moved to GPU; otherwise it runs on CPU. Labels are normalised to entity types by lowercasing and replacing spaces with underscores (e.g. "earnings announcement" → "earnings_announcement").

Configuration:

• GLINER_MODEL_NAME — HuggingFace model identifier
• GLINER_MNPI_THRESHOLD — Minimum confidence for predictions (default: 0.5)

If the gliner package is not installed, detect() returns an empty list gracefully.

---

Tier 3: DeBERTa Contextual Validation

Module: backend/app/core/dlp_deberta.py

Tier 3 is a validator, not a new detector. It receives the match candidates produced by Tiers 1 and 2 and uses contextual NLI analysis to distinguish true positives from false positives.

Current production model: deberta-dlp-v4 (DeBERTa-v3-large, 304M params, 99.48% eval accuracy). A future model version (v6) is planned. See DeBERTa Tier 3 Admin Guide for model details and deployment.

How it works

DeBERTaValidatorClient.validate() sends existing DLPMatch objects to the DeBERTa microservice at POST /validate:

payload = {
    "matches": [
        {
            "text": m.matched_text,
            "entity_type": m.entity_type,
            "start": m.start, "end": m.end,
            "score": m.confidence,
        }
    ],
    "context": full_scanned_text,
    "threshold": optional_threshold,
}

The microservice returns validated_matches with is_true_positive and contextual_score fields. The client applies the results:

• True positive: Match confidence is updated to contextual_score.
• False positive: Match confidence is demoted to 0.1 (_DEMOTED_CONFIDENCE), falling below any practical pipeline confidence_threshold.

Matches not present in the response are left unchanged.

Confidence threshold routing

The route_l3_result() function in dlp.py maps the softmax confidence returned by the microservice to an enforcement decision using three zones:

Confidence zone	Range	Routing
Hard block	P > 0.70	Blocked regardless of org sensitivity
Ambiguous	0.35 ≤ P ≤ 0.70	Depends on dlp_sensitivity setting
Pass	P < 0.35	No flag; request proceeds

Ambiguous zone routing by dlp_sensitivity:

dlp_sensitivity	Result
"high"	soft_block, audit_flag=True, path: l3_soft_block_high_sensitivity
"standard"	pass, audit_flag=True, path: l3_elevated_flag_standard

Every scan that produces matches emits a structured dlp_l3_routing log entry recording l3_confidence, routing_path, and audit_flag for the audit trail.

Default endpoint: http://deberta-validator:8201. Override with DLP_DEBERTA_VALIDATOR_URL.

Circuit breaker: Same pattern as the NER microservice — 3 failures opens the circuit for 60 seconds.

Fail mode: Same DLP_INFERENCE_FAIL_MODE env var. Default "closed" blocks on microservice failure.

What Tier 3 catches that Tiers 1 and 2 miss

• Context-dependent false positives: A 16-digit number in a paragraph about football statistics is not a credit card. DeBERTa can read the surrounding sentence and demote the confidence, preventing a false block.
• Obfuscated or partially encoded secrets: Text that rephrases or describes a credential without using its exact format.
• Semantic PII: Names or addresses embedded in natural-language text where structure-based detectors are unreliable.

---

Tier 4: CredInt Credential Intelligence

Module: backend/app/services/dlp.py (run_credint_check, route_credint_result), backend/app/core/dlp_microservice.py (CredentialIntelligenceClient)

Tier 4 checks whether credential candidates in the text appear in a database of known-compromised credentials. It runs concurrently with the L1→L2→L3 chain as an asyncio.create_task, so its latency does not add to the serial DLP scan time.

Architecture

The check has two phases:

1. L1 extraction (credential_extractor.extract_credential_candidates) — A fast synchronous pass that finds credential-shaped tokens in the text (API key patterns, user:password pairs, key=value assignments). The actual credential values are never logged; only SHA-1 prefixes appear in audit records.

2. Parallel CredInt lookups — asyncio.gather() fans out one CredentialIntelligenceClient.check(candidate.value) call per candidate. The service implements k-anonymity (HIBP-style): only the first 5 hex characters of the SHA-1 hash are sent over the wire; the full hash is never transmitted.

Frequency buckets and routing

Each hit result from the CredInt service includes a frequency_bucket field. The route_credint_result() function maps this to an action decision:

Bucket	dlp_sensitivity	Action	Audit flag
critical or high	"high"	soft_block	Yes
critical or high	"standard"	pass	Yes (elevated confidence flag)
medium or low	Either	pass	Yes
No hit	—	pass	No

Confidence mapping: critical/high → 1.0, medium/low → 0.5, no hit → 0.0.

Audit fields

run_credint_check() returns a dict that is merged into the DLP audit log extra_metadata:

Field	Type	Description
credint_enabled	bool	Always True when CredInt is enabled for the org
credint_hit	bool | None	Whether any candidate matched
frequency_bucket	str | None	Highest-severity bucket across all hits
context_type	str | None	Context type of the first hit candidate
sha1_prefix	str | None	SHA-1 prefix of the first hit (never full hash)
credint_confidence	float	Routing confidence: 1.0 / 0.5 / 0.0
credint_action	str	soft_block or pass
credint_audit_flag	bool	Whether routing elevated the audit flag
credint_routing_path	str	Routing branch identifier
credint_available	bool	Whether the service responded
candidate_count	int	Number of credential candidates extracted

Fail-open behavior

Any exception in run_credint_check() is caught and logged. The returned dict defaults to safe no-hit values (credint_available=False, credint_action="pass"). A missing or unavailable CredInt service never blocks a request.

---

Detection Capabilities by Tier

Entity type	Tier 0 (TF-IDF)	Tier 1 (Regex)	Tier 2 (NER)	Tier 3 (DeBERTa)	Tier 4 (CredInt)
Prompt injection	Yes — binary classifier, short-circuit	—	—	—	—
Cloud API keys (AWS, GCP, Azure)	—	Yes — structural patterns, high confidence	—	Contextual validation	Corpus lookup
AI/ML tokens (Anthropic, OpenAI, HuggingFace)	—	Yes	—	Contextual validation	Corpus lookup
PEM/SSH private keys	—	Yes — header pattern	—	Contextual validation	—
Database connection strings	—	Yes — URI pattern	—	Contextual validation	Corpus lookup
Auth tokens (GitHub PAT, JWT, Bearer)	—	Yes	—	Contextual validation	Corpus lookup
Credit cards / bank identifiers	—	Yes — Luhn validated	—	Contextual validation	—
US SSN / ITIN / international IDs	—	Yes — with structural validators	Contextual NER	Contextual validation	—
Passports / driver licenses	—	Yes	—	Contextual validation	—
Person names	—	—	Yes (spaCy PERSON, GLiNER)	Contextual validation	—
Organisation names	—	—	Yes (spaCy ORG)	Contextual validation	—
Locations / addresses	—	—	Yes (spaCy GPE, LOC)	Contextual validation	—
Financial amounts	—	—	Yes (spaCy MONEY)	Contextual validation	—
MNPI (earnings, M&A, insider info)	—	—	Yes (GLiNER zero-shot)	Contextual validation	—
Healthcare (DEA, NPI, MRN, NDC)	—	Yes	—	Contextual validation	—
Compromised credentials	—	—	—	—	Platform: remote HTTP service (CredentialIntelligenceClient); Outpost: local bloom filter
Obfuscated / context-dependent PII	—	—	Partial	Yes — primary detection	—

---

Redaction Modes

When a scan returns a REDACT action, matched spans are replaced in-line before the text is passed downstream or returned to the caller. The original text is preserved in the audit record.

In the platform (OutputScanner)

OutputScanner.get_redacted_chunk() applies redactions by processing matches in reverse position order to preserve character offsets:

result = original
sorted_matches = sorted(scan_result.matches, key=lambda m: m.start, reverse=True)
for match in sorted_matches:
    result = result[:match.start] + "[REDACTED]" + result[match.end:]

The redaction token is [REDACTED] (entity type not included in the platform output scanner).

In the outpost

redact_text() in outpost/dlp/pipeline.py uses the entity type in the marker:

result = result[:start] + f"[REDACTED:{entity_type}]" + result[end:]

This produces tokens like [REDACTED:ssn], [REDACTED:credit_card], [REDACTED:email].

Action hierarchy

When multiple rules match the same text, the highest-severity action wins:

block > cancel > redact > log_only

This is implemented in _resolve_action() in the outpost pipeline and in DLPPipeline action mapping in the platform core.

---

Custom Pattern Authoring

Modules: backend/app/core/org_dlp_layer.py, backend/app/services/org_dlp_rules.py, backend/app/models/org_dlp_rule.py

Organisations can extend or restrict the default detection rules without modifying platform defaults. All org rules are stored in the org_dlp_rules table and cached per-org with a 60-second TTL (load_org_rules_cached).

Rule types

rule_type	Purpose
custom_pattern	Adds a new regex pattern that runs alongside platform defaults
suppress_default	Disables a specific platform default pattern for the org

OrgDLPRule model fields

Field	Description
org_id	Owning organisation (tenant isolation enforced in all queries)
rule_type	"custom_pattern" or "suppress_default"
name	Human-readable rule name (also used as suppression key by name match)
pattern	Regex pattern string (required for custom_pattern, null for suppress_default)
target_rule_id	Platform rule UUID or name to suppress (for suppress_default)
enabled	Whether the rule is active
action_tier	"log_only", "redact", "block", or "prompt" (default: "log_only")
custom_entity_type	Entity type label for matches (default: "org_custom_pattern")

OrgDLPLayer integration

OrgDLPLayer is a composable wrapper that loads org rules from the cache and provides two operations:

layer = OrgDLPLayer(org_id)
await layer.load(db)

# 1. Get suppressed platform pattern names for post-scan filtering
suppressed = layer.get_suppressed_pattern_names()

# 2. Run custom patterns alongside the platform scan
org_matches = await layer.scan_custom_patterns(text)

Custom patterns are compiled using safe_compile() (from backend.app.core.safe_regex), which guards against ReDoS by enforcing a regex timeout. Invalid or dangerous patterns are logged and skipped.

filter_platform_matches() removes any DLPMatch whose detector_name is in the suppression set:

filtered = [m for m in matches if m.detector_name not in self._suppressed_names]

Custom pattern matches get a detector_name in the format org_custom:{rule_name}:{action_tier} with confidence=1.0 (deterministic regex match).

Audit trail

Every create, update, delete, enable, or disable action on an OrgDLPRule writes an immutable OrgDLPRuleAudit entry capturing the actor, the action, and before/after state snapshots in JSONB. The cache is invalidated immediately on any mutation via invalidate_org_rules_cache(org_id).

---

DLP Policy Integration

Compliance bundles and entity type filtering

filter_scan_result_by_entity_types() post-processes scan results to keep only matches whose entity types are covered by active compliance bundles. This lets orgs activate specific detection categories (e.g. HIPAA, PCI-DSS) without enabling all detectors globally.

get_active_entity_types() resolves the set of active entity types from the compliance bundle configuration, supporting two enforcement modes:

Mode	Behavior
additive (default)	User’s group bundle entity types are merged with globally active bundle types
strict	Only the user’s group-assigned bundle entity types are enforced

When no compliance bundles exist, get_active_entity_types() returns None, which disables bundle filtering and runs all detectors unrestricted.

Webhook integration

fire_dlp_trigger() fires a dlp_trigger webhook event whenever a scan produces a BLOCK or REDACT action. The webhook runs in a background asyncio.create_task with its own database session, so delivery never blocks the primary scan flow. One webhook fires per OutputScanner session (subsequent BLOCK/REDACT scans in the same stream are suppressed to prevent spam).

Webhook payload:

{
  "conversation_id": "...",
  "user_id": "...",
  "entity_type": "api_key",
  "action": "BLOCK",
  "detector_name": "aws_access_key_id"
}

DLP events

Every BLOCK, REDACT, or LOG_ONLY detection is recorded in the dlp_events table (DLPEvent model) by record_dlp_event(). Severity is auto-calculated:

Action	Severity	Exception
block	high	api_key, api_secret, secret_key entity types → critical
redact	medium	—
log_only	low	—

Events support a lifecycle workflow: detected → investigating → resolved or false_positive.

---

Outpost DLP

The Hybrid Outpost runs the same 5-tier pipeline entirely within the customer’s environment. Customer text never leaves the VPC for DLP decisions.

Outpost pipeline (outpost/dlp/pipeline.py)

DLPPipeline (outpost) is structurally identical to the platform pipeline:

0. Tier 0 (TF-IDF) — Tier0Classifier using a local tier0_model.joblib. Runs on input only. Configurable via policy bundle overrides (tier0_prompt_injection_enabled, tier0_confidence_threshold). Fails open on classification error (unlike the platform, which fails closed).
1. Tier 1 (regex) — RegexScanner using built-in patterns plus policy-bundle patterns loaded from the cached policy sync. Supports compiled Rust backend via DLP_SCANNER_BACKEND=rust (same CompiledScannerCache as platform, with identical pattern conversion and post-validation).
2. Tier 2 (NER) — NERScanner using spaCy (en_core_web_sm by default). Controlled by DLP_NER_ENABLED / dlp_ner_enabled.
3. Tier 3 (DeBERTa) — DeBERTaScanner using a locally bundled ONNX model. Auto-activates when DEBERTA_MODEL_PATH is set and the file exists. Supports two loading strategies: optimum.onnxruntime.ORTModelForSequenceClassification (preferred) or raw onnxruntime.InferenceSession (minimal-dependency fallback).
4. Tier 4 (CredInt) — CredIntScanner using an on-disk bloom filter loaded from CREDINT_BLOOM_PATH. Entirely in-process; zero network calls at scan time.

Outpost-specific behaviors

Early termination: If Tier 1 finds a BLOCK-action match, the pipeline returns immediately. Tiers 2 through 4 do not run.

Policy rule overlay: update_policy_rules() receives DLP rules from the policy bundle sync. Rules override default entity type action tiers and support direction filtering ("input", "output", or "both").

Entity deduplication: _deduplicate_entities() removes overlapping spans, preferring higher-confidence matches. When confidence is equal, tier priority wins: deberta > ner > regex.

Redaction format: redact_text() uses [REDACTED:{entity_type}] markers.

Offline capability: The outpost continues enforcing the last-cached policy pack when the Cloud control plane is unreachable. The CredInt bloom filter is loaded from the local disk and never requires a network call at scan time. DeBERTa inference is fully local (ONNX Runtime, no external service).

Tier 3 auto-activation: DeBERTaScanner.is_available re-checks the model file path on each call if the model was not present at startup. When a model file is dropped into the configured path, Tier 3 activates on the next scan without a restart.

Bloom filter CDN refresh: CredIntScanner.refresh_from_cdn() supports hot-swapping the bloom filter from a CDN URL using conditional HTTP GET (ETag). The swap is atomic (os.replace()). A version guard prevents downgrading to an older snapshot date.

Outpost configuration keys

Environment variable	Default	Description
DLP_ENABLED	true	Enable the DLP pipeline entirely
tier0_prompt_injection_enabled	true	Enable Tier 0 TF-IDF prompt injection pre-filter
tier0_confidence_threshold	0.95	Tier 0 minimum confidence to block
tier0_model_path	outpost/data/tier0_model.joblib	Path to the serialized Tier 0 model
DLP_SCANNER_BACKEND	python	Tier 1 backend: python (re module) or rust (compiled scanner). Falls back to Python if Rust extension not installed.
DLP_NER_ENABLED	true	Enable Tier 2 spaCy NER
DLP_NER_MODEL	en_core_web_sm	spaCy model for NER
DLP_NER_DEVICE	auto	Device selection: auto, cpu, cuda
DLP_DEBERTA_ENABLED	false	Enable Tier 3 DeBERTa contextual classifier
DEBERTA_MODEL_PATH	—	Path to the ONNX model file (model.onnx)
CREDINT_BLOOM_PATH	—	Path to the bloom filter binary (.arbf format)
CREDINT_KANON_ENABLED	false	Enable k-anonymity secondary check for bloom hits
CREDINT_CDN_URL	—	CDN URL for background bloom filter refresh

---

Per-Tier OTel Metrics

ScanEngine emits OpenTelemetry metrics for each pipeline tier, enabling per-tier observability in Grafana dashboards and alerting.

Tier 0 metrics

Metric	Type	Labels	Description
tier0_decisions_total	Counter	decision (block/pass), confidence_bucket (high/medium/low)	Classification decisions
tier0_classify_fail_closed_total	Counter	—	Fail-closed events (platform only)
tier0_latency_ms	Histogram	—	Classification latency

Tier 1 metrics

Metric	Type	Labels	Description
dlp_tier1_scan_total	Counter	backend (python/rust), action (block/redact/log_only/none)	Scan decisions
dlp_tier1_latency_ms	Histogram	backend	Scan latency
dlp_tier1_matches_total	Counter	entity_type, action_tier	Match counts by entity type

Tier 2 metrics

Metric	Type	Labels	Description
dlp_tier2_scan_total	Counter	detector (ner/presidio/gliner), action	Scan decisions
dlp_tier2_latency_ms	Histogram	detector	Per-detector latency
dlp_tier2_circuit_breaker_state	Gauge	detector	Circuit breaker state (0=closed, 1=open, 2=half-open)

Tier 3 metrics

Metric	Type	Labels	Description
dlp_tier3_validation_total	Counter	result (confirmed/demoted/unchanged)	Validation decisions
dlp_tier3_latency_ms	Histogram	—	Validation round-trip latency
dlp_tier3_skip_total	Counter	reason (high_confidence/validator_confirmed)	Skipped validations
dlp_tier3_circuit_breaker_state	Gauge	—	Circuit breaker state

Tier 4 metrics

Metric	Type	Labels	Description
dlp_credint_check_total	Counter	result (hit/miss/error), bucket (critical/high/medium/low)	CredInt lookup results
dlp_credint_latency_ms	Histogram	—	Lookup latency
dlp_credint_candidates_total	Counter	—	Credential candidates extracted

Pipeline-level metrics

Metric	Type	Labels	Description
dlp_scan_total	Counter	direction (input/output), action (block/redact/log_only/none)	End-to-end scan decisions
dlp_scan_latency_ms	Histogram	direction	End-to-end pipeline latency
dlp_early_termination_total	Counter	tier	Early terminations by tier

For OTel configuration, see OTel Configuration Guide.

---

Performance Considerations

Per-tier latency profile

Tier	Typical latency	Notes
Tier 0 — TF-IDF	< 1 ms	Synchronous sklearn predict_proba. Input only.
Tier 1 — Regex	< 1 ms (p50) for 1 KB	Synchronous, no I/O. Scales linearly with text length.
Tier 2 — NER microservice	10–50 ms typical	Network round-trip to GPU node pool. Circuit breaker opens after 3 failures.
Tier 2 — GLiNER	50–200 ms (CPU)	Lazy-loaded singleton; GPU-accelerated when CUDA available.
Tier 3 — DeBERTa validator	20–100 ms	Depends on number of candidates and GPU/CPU availability.
Tier 4 — CredInt	Concurrent; < 5 ms bloom filter check	Hash computation only. k-anonymity HTTP call adds ~50 ms when enabled.

The outpost benchmark (benchmarks/dlp_pipeline_bench.py) measures end-to-end pipeline latency at three payload sizes with NER enabled and DeBERTa disabled:

Payload	E2E mean	P95
1 KB	< 2 ms	—
10 KB	5–15 ms	—
100 KB	30–80 ms	—

(Actual values depend on CPU speed and spaCy model size.)

Streaming output scan interval

OutputScanner triggers a scan every chunk_interval characters of accumulated output (default: 500). Reducing this value catches sensitive data earlier in the stream but increases scan frequency and total GPU utilisation. Increasing it reduces overhead at the cost of later detection.

Early termination

The most significant optimisation is early exit: a BLOCK-action match at any tier short-circuits the remaining pipeline. ScanEngine checks the resolved action after each tier completes — if the action is BLOCK, remaining tiers are skipped and the result is returned immediately. For texts containing high-confidence structured secrets (AWS keys, PEM headers), the pipeline rarely advances past Tier 1. Early termination events are tracked by the dlp_early_termination_total OTel counter with a tier label.

Batch processing

For bulk scanning outside the hot path, call DLPPipeline.scan() directly with pre-batched text. The Tier 1 RegexDetector is safe to call from multiple threads (compiled patterns are read-only). Tier 2 and Tier 3 are async-safe but not thread-safe — use a single event loop per pipeline instance.

---

Configuration Reference

Platform environment variables

Variable	Default	Description
DLP_OUTPUT_SCANNING_ENABLED	true	Enable output scanning on model responses
DLP_SCANNER_BACKEND	python	Tier 1 backend: python (re module) or rust (compiled scanner)
DLP_NER_MICROSERVICE_URL	http://ner-gpu:8200	NER GPU microservice base URL
DLP_NER_MICROSERVICE_TIMEOUT	5.0	HTTP timeout for NER requests (seconds)
DLP_DEBERTA_VALIDATOR_URL	http://deberta-validator:8201	DeBERTa validator microservice base URL
DLP_DEBERTA_VALIDATOR_TIMEOUT	5.0	HTTP timeout for DeBERTa validation requests (seconds)
DLP_INFERENCE_FAIL_MODE	closed	"closed" blocks on microservice failure; "open" passes through unscanned
CREDINT_SERVICE_URL	—	CredInt microservice URL for platform-side credential lookups
compliance_enforcement_mode	additive	Bundle enforcement mode: "additive" or "strict"

DLP sensitivity

dlp_sensitivity is an org-level setting (not an environment variable). It controls routing in both the DeBERTa ambiguous zone and the CredInt high-frequency bucket:

• "standard" (default) — passes ambiguous L3 signals with an elevated audit flag.
• "high" — promotes ambiguous L3 signals and high-frequency CredInt hits to soft_block.

Org rule cache TTL

The per-org rule cache TTL is _CACHE_TTL_SECONDS = 60.0 in org_dlp_rules.py. The cache is invalidated immediately on any CRUD mutation. In high-frequency scanning scenarios, the 60-second TTL means a newly added org rule may not take effect for up to one minute on instances that have already cached the previous rule set.

---

See also

• Request Lifecycle — Where DLP fits in the 7-stage gateway pipeline
• DLP API — Rule management endpoints, action tiers, and L3 confidence threshold reference
• Outpost CLI — Deploying and configuring the Hybrid Outpost DLP pipeline
• CredInt Deployment Guide — Bloom filter format, CDN refresh, and k-anonymity setup