DLP Pipeline Configuration

The Arbitex DLP pipeline inspects every prompt and model response before it reaches its destination. This guide explains how to configure each tier, define custom detection rules, map findings to policy actions, and test your configuration without affecting production traffic.

Configuration requires the Org Admin role.

---

Architecture overview

The pipeline runs five tiers sequentially. A finding from an earlier tier does not stop later tiers — all tiers run and their results are merged before the policy engine evaluates actions.

Request prompt
      │
      ▼
┌─────────────────────────────┐
│  Tier 1: Regex matching      │  ~1 ms    70+ built-in patterns + custom rules
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  Tier 2: NER (spaCy)         │  ~20 ms   Named entity recognition (en_core_web_sm)
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│  Tier 3: DeBERTa NLI         │  ~150 ms  Contextual validation of ambiguous hits
└─────────────┬───────────────┘
              │
              ▼
        Policy engine
    (ALLOW / REDACT / BLOCK)
              │
              ▼
        Model provider

Tiers 1 and 2 are always active. Tier 3 is configurable per organization and runs only on traffic that passes Tiers 1 and 2 without a definitive finding — keeping latency low for clean traffic.

---

Tier 1: Regex matching

Built-in patterns

Arbitex ships 70+ platform patterns covering:

Category	Examples
Financial	Credit card (Luhn-validated), IBAN+BIC, routing numbers, SWIFT
Government IDs	US SSN, EIN, passport formats (15+ countries)
Health	US NPI, DEA numbers, ICD-10 patterns
Cloud credentials	AWS keys, GCP service accounts, Azure SAS tokens
Generic secrets	Bearer tokens, JWTs, private key headers, connection strings
Contact	Email, US/international phone, postal codes

To list all available platform patterns:

curl "https://api.arbitex.ai/api/v1/admin/dlp-rules?source=platform&limit=100" \
  -H "Authorization: Bearer $ARBITEX_API_KEY"

Each pattern entry includes name, rule_type (regex), pattern, enabled, and action_tier (which tier triggers the action).

Disabling a built-in pattern

If a built-in pattern generates false positives for your workload, disable it at the org level without deleting the platform rule:

# First, get the platform rule ID you want to suppress
RULE_ID="dlprule_01HZ..."

# Create an org-level override that disables it
curl -X POST "https://api.arbitex.ai/api/orgs/{org_id}/dlp-rules" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rule_type": "regex",
    "name": "suppress-phone-false-positives",
    "target_rule_id": "'$RULE_ID'",
    "enabled": false
  }'

The org-level override takes precedence. The platform default is suppressed only for your organization.

Creating a custom regex rule

Add a custom pattern for data types specific to your organization — internal account numbers, proprietary identifiers, or industry-specific formats:

curl -X POST "https://api.arbitex.ai/api/v1/admin/dlp-rules" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "internal-employee-id",
    "rule_type": "regex",
    "pattern": "\\bEMP-[0-9]{6}\\b",
    "custom_entity_type": "EMPLOYEE_ID",
    "action_tier": 1,
    "enabled": true
  }'

Field	Description
name	Human-readable rule name (must be unique within org)
rule_type	"regex", "ner", or "gliner"
pattern	Python re-compatible regex. Tested with a 1-second timeout per request
custom_entity_type	Label applied to findings from this rule (appears in audit log and DLP events)
action_tier	Which pipeline tier the finding is attributed to for action mapping
enabled	true to activate immediately

Pattern best practices:

• Use word boundary anchors (\b) to avoid partial matches
• Test your pattern with the DLP test endpoint before enabling in production (see Testing rules)
• Avoid catastrophic backtracking — the gateway enforces a 1-second regex timeout; rules that exceed it are disabled automatically

Testing a regex pattern

Before saving a rule, test it against sample text:

curl -X POST "https://api.arbitex.ai/api/v1/admin/dlp-rules/test" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rule_type": "regex",
    "pattern": "\\bEMP-[0-9]{6}\\b",
    "text": "Please update EMP-042891 employee record with new address."
  }'

{
  "matches": [
    {
      "match": "EMP-042891",
      "start": 14,
      "end": 24,
      "confidence": 1.0
    }
  ],
  "elapsed_ms": 0.4
}

A matches array with at least one entry confirms the pattern works. elapsed_ms shows the regex execution time — keep this well under 100 ms to ensure safe production performance.

---

Tier 2: NER (Named Entity Recognition)

Tier 2 uses the spaCy en_core_web_sm model for named entity recognition to identify unstructured PII in free text that doesn’t match fixed regex patterns — person names, organization names, locations, monetary values, and dates.

Default NER entity types

The following spaCy label mappings are active by default. Arbitex canonical entity type names (used in policy rules and audit logs) are shown alongside the spaCy source labels.

spaCy label	Arbitex entity type	Examples
PERSON	pii_name	“John Smith”, “Dr. Sarah Chen”
ORG	org_name	“Acme Corporation”, “First National Bank”
GPE	location	“London”, “Springfield, IL”
MONEY	financial_amount	“$1,500”, “€2,000”
DATE	date	“March 3, 1985”, “next Tuesday”

Note

Phone numbers, email addresses, dates of birth, and medical record references are not detected by the spaCy NER tier. These require either Tier 1 regex rules (for structured patterns like phone numbers and email addresses) or the Tier 3 DeBERTa model (for contextual medical content). See Tier 1: Regex matching for adding custom regex patterns.

NER confidence threshold

spaCy NER detections are reported with a fixed moderate confidence score. To filter findings by confidence, set entity_confidence_min on policy rules that use the entity_types condition. See the Policy Engine reference for condition syntax.

---

Tier 3: DeBERTa contextual validation

Tier 3 runs a DeBERTa NLI (Natural Language Inference) model on content that passes Tiers 1 and 2 without a definitive finding. It validates whether ambiguous content is genuinely sensitive in context.

Sensitivity modes

DeBERTa operates in one of two sensitivity modes:

Mode	Ambiguous score range (0.35–0.70)	Behavior
standard (default)	Pass	Audit flag raised, no block
high	Soft block	Request rejected with dlp_soft_block

A definitive DeBERTa score (above 0.70 or below 0.35) always triggers a block or pass regardless of sensitivity mode.

Set your organization’s sensitivity:

curl -X PATCH "https://api.arbitex.ai/api/v1/admin/orgs/{org_id}/dlp-config" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dlp_sensitivity": "high"
  }'

Use "high" for organizations handling regulated data (HIPAA, PCI-DSS, GDPR) where false negatives are more costly than false positives. Use "standard" for general-purpose workloads where latency and user experience take priority.

Enabling Credential Intelligence (CredInt)

CredInt runs in parallel with Tier 3. It checks detected credentials against a breach corpus of known-compromised secrets. When enabled, a credential that matches a known-breached value receives elevated severity regardless of DeBERTa’s confidence score.

curl -X PATCH "https://api.arbitex.ai/api/v1/admin/orgs/{org_id}/dlp-config" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "credint_enabled": true
  }'

Under high sensitivity, a CredInt critical or high frequency bucket hit triggers a soft block. Under standard sensitivity, it raises an elevated audit flag without blocking.

See Credential Intelligence for full details.

---

Policy action mapping

DLP findings do not directly block requests — they produce findings that the policy engine evaluates. You map DLP finding types to actions using policy rules.

Action types

Action	Description
ALLOW	Finding is logged; request proceeds unmodified
REDACT	Matched text is replaced with [REDACTED-{TYPE}] before forwarding
BLOCK	Request is rejected with HTTP 400
REQUIRE_APPROVAL	Request is held for human review; requester receives a pending status

Example: block SSN in prompts

Create a policy rule that blocks any prompt containing an SSN finding:

curl -X POST "https://api.arbitex.ai/api/v1/admin/orgs/{org_id}/policy-rules" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "block-ssn-in-prompt",
    "description": "Block requests where the prompt contains a Social Security Number",
    "conditions": {
      "dlp_finding_types": ["SSN"],
      "dlp_locations": ["prompt"]
    },
    "action": "BLOCK",
    "enabled": true
  }'

Example: redact credit card numbers in responses

Redact credit card numbers from model responses before they reach the user:

curl -X POST "https://api.arbitex.ai/api/v1/admin/orgs/{org_id}/policy-rules" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "redact-cc-in-response",
    "description": "Redact credit card numbers from model responses",
    "conditions": {
      "dlp_finding_types": ["CREDIT_CARD"],
      "dlp_locations": ["response"]
    },
    "action": "REDACT",
    "enabled": true
  }'

Example: require approval for medical records

Route requests containing medical identifiers to a human reviewer:

curl -X POST "https://api.arbitex.ai/api/v1/admin/orgs/{org_id}/policy-rules" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "approve-medical-record-access",
    "description": "Hold requests containing NPI or medical record references for approval",
    "conditions": {
      "dlp_finding_types": ["NPI", "MEDICAL_RECORD"]
    },
    "action": "REQUIRE_APPROVAL",
    "enabled": true
  }'

See Policy Engine user guide and Policy Engine API reference for full rule schema documentation.

---

Testing with the policy simulator

Before deploying rule changes to production, use the policy simulator to preview what will happen to a given request:

curl -X POST "https://api.arbitex.ai/api/v1/admin/policy/simulate" \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Please update employee EMP-042891. Their SSN is 123-45-6789.",
    "model": "anthropic/claude-sonnet-4-20250514",
    "user_id": "usr_test_alice"
  }'

{
  "outcome": "BLOCK",
  "dlp_findings": [
    {
      "tier": 1,
      "type": "EMPLOYEE_ID",
      "match": "EMP-042891",
      "confidence": 1.0,
      "location": "prompt"
    },
    {
      "tier": 1,
      "type": "SSN",
      "match": "123-45-6789",
      "confidence": 1.0,
      "location": "prompt"
    }
  ],
  "policy_rules_evaluated": [
    {
      "rule_id": "pr_01HZ...",
      "name": "block-ssn-in-prompt",
      "matched": true,
      "action": "BLOCK"
    },
    {
      "rule_id": "pr_02HZ...",
      "name": "block-pii-in-prompt",
      "matched": true,
      "action": "BLOCK"
    }
  ],
  "effective_action": "BLOCK",
  "simulation_only": true
}

The simulator runs the full DLP pipeline and policy evaluation without making any real requests to the model or writing to the audit log. simulation_only: true confirms no production traffic was affected.

Use the simulator to:

• Verify a new rule catches what you expect before enabling it
• Confirm that disabling a rule doesn’t leave a gap in coverage
• Test edge cases and boundary conditions with sample inputs
• Train your team on how the pipeline behaves

See Outpost Policy Simulator for simulator usage from outpost deployments.

---

Viewing effective rules

To see the merged set of rules actually applied to your organization (platform defaults + org overrides):

curl "https://api.arbitex.ai/api/orgs/{org_id}/dlp-rules/effective" \
  -H "Authorization: Bearer $ARBITEX_API_KEY"

{
  "rules": [
    {
      "id": "dlprule_platform_01",
      "source": "platform",
      "name": "credit-card-luhn",
      "rule_type": "regex",
      "enabled": true,
      "action_tier": 1,
      "custom_entity_type": "CREDIT_CARD"
    },
    {
      "id": "dlprule_org_01HZ",
      "source": "org",
      "name": "internal-employee-id",
      "rule_type": "regex",
      "enabled": true,
      "action_tier": 1,
      "custom_entity_type": "EMPLOYEE_ID"
    }
  ],
  "total": 67
}

The source field shows whether each rule comes from the platform defaults or your org-level configuration.

---

Rule versioning

Every change to a DLP rule creates an immutable version record. View the history of a rule:

curl "https://api.arbitex.ai/api/v1/admin/dlp-rules/{rule_id}/versions" \
  -H "Authorization: Bearer $ARBITEX_API_KEY"

{
  "versions": [
    {
      "version": 2,
      "changed_by": "[email protected]",
      "changed_at": "2026-03-12T14:00:00Z",
      "change_type": "update",
      "old_pattern": "\\bEMP-[0-9]{5}\\b",
      "new_pattern": "\\bEMP-[0-9]{6}\\b",
      "reason": "Employee IDs extended to 6 digits"
    },
    {
      "version": 1,
      "changed_by": "[email protected]",
      "changed_at": "2026-03-01T09:00:00Z",
      "change_type": "create",
      "new_pattern": "\\bEMP-[0-9]{5}\\b"
    }
  ]
}

See DLP Rules API reference for bulk export, import, and full version history APIs.

---

Scanner backend selection

The DLP_SCANNER_BACKEND environment variable controls which Tier 1 (regex) scanning engine is used.

Backend	Value	Description
Python (default)	python	Pure-Python re module scanning. No additional dependencies.
Rust (compiled)	rust	Compiled multi-pattern automaton via CompiledScannerCache from arbitex_core. Higher throughput for regex matching.

Set the variable in your deployment configuration:

# .env or environment config
DLP_SCANNER_BACKEND=rust

Fallback behavior: If DLP_SCANNER_BACKEND=rust is set but the compiled Rust extension (_dlp_scanner_rs) is not installed, the platform logs a warning (dlp_rust_scanner_fallback) and falls back to the Python backend transparently. There are zero functional differences between the two backends — the Rust scanner is a drop-in performance optimization.

Outpost parity: The Outpost supports the same DLP_SCANNER_BACKEND flag. Both the platform RegexDetector and Outpost RegexScanner use the identical CompiledScannerCache and apply the same Python-side post-validation (Luhn, IBAN mod-97, ITIN group ranges).

---

NER gate configuration

The NER gate controls whether Tier 2 NER runs during DLP scanning. NER detects person names, organizations, locations, and other natural-language entities that structural regex patterns miss — but adds 10–50 ms of latency per scan.

DLP_NER_FORCE system config key

The DLP_NER_FORCE key in system_config controls NER behavior globally. Per-org overrides use the key dlp_ner_force:{org_id}.

Value	Behavior
auto (default)	NER runs only when the org’s enabled compliance packs require it and trigger keywords are present in the text
always	NER always runs regardless of pack configuration
never	NER is never invoked — only Tier 1 (regex) and Tier 3 (DeBERTa) run

View and set the global NER mode

# Check current NER force setting
curl -s -X GET \
  "https://api.arbitex.ai/api/v1/admin/config/DLP_NER_FORCE" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Set NER force to "always"
curl -s -X PUT \
  "https://api.arbitex.ai/api/v1/admin/config/DLP_NER_FORCE" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": "always"}'

Per-org NER override

Override the global setting for a specific organization:

# Set per-org NER force override
curl -s -X PUT \
  "https://api.arbitex.ai/api/v1/admin/config/dlp_ner_force:${ORG_ID}" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": "never"}'

# Remove per-org override (falls back to global setting)
curl -s -X DELETE \
  "https://api.arbitex.ai/api/v1/admin/config/dlp_ner_force:${ORG_ID}" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

When to adjust

Scenario	Recommended setting
Org handles HIPAA/GDPR/CCPA data with person names	auto (default) — compliance packs flag NER as required
Org uses PCI-DSS/SOX only (structural patterns)	auto — NER is skipped automatically
Maximum detection sensitivity regardless of latency	always
Ultra-low-latency scanning (code-only workloads)	never — Tier 1 regex only

NER gate resolution is cached for 60 seconds. After changing the NER force setting, allow up to 60 seconds for all platform instances to pick up the change.

---

Outpost DLP configuration

The Hybrid Outpost runs the same 5-tier pipeline entirely within the customer’s environment. All configuration is applied through the Outpost’s environment variables and policy bundle sync.

Outpost environment variables

Variable	Default	Description
DLP_SCANNER_BACKEND	python	Tier 1 backend: python or rust
DLP_NER_ENABLED	true	Enable/disable Tier 2 NER
DEBERTA_MODEL_PATH	—	Path to ONNX model file for Tier 3
CREDINT_BLOOM_PATH	—	Path to CredInt bloom filter file for Tier 4

Policy rule overlay

DLP rules sync from the platform to the Outpost via the policy bundle. Rules support direction filtering:

Direction	Scans
input	User prompts only
output	Model responses only
both	Both input and output

---

Reference

Resource	Link
DLP event monitoring	DLP Event Monitoring
DLP Rules API	API Reference: DLP Rules
Policy Engine user guide	Policy Engine User Guide
Policy Engine API	API Reference: Policy Engine
Policy simulator	Outpost Policy Simulator
Credential Intelligence	Credential Intelligence