DLP confidence calibration

Arbitex DLP confidence scores express the pipeline’s certainty that a detected pattern is a genuine sensitive data match and not a false positive. Understanding how confidence scores are generated, boosted, and filtered lets you tune DLP detection to your organization’s needs.

---

What confidence scores represent

A DLP confidence score is a decimal value from 0.0 to 1.0 representing the pipeline’s estimated probability that a detected pattern is a genuine sensitive data match.

Score range	Interpretation
0.90–1.00	Near-certain match — almost all patterns at this level are genuine
0.70–0.89	Strong match — very few false positives expected
0.50–0.69	Probable match — expect occasional false positives
0.35–0.49	Ambiguous — may be genuine or contextual noise
0.00–0.34	Low confidence — likely false positive

Note

Confidence scores in Arbitex DLP represent empirical precision at each tier — not Bayesian probabilities. A score of 0.90 means that 90% of similarly-scored matches in calibration testing were genuine sensitive data. Scores are calibrated against a validation set for each entity type and may be recalibrated as the pattern library evolves.

Confidence scores are not uniform across entity types. Secrets (API keys, tokens, bearer tokens) ship with high baseline confidence (0.95+) because high-entropy patterns have very low false-positive rates. Phone numbers and email addresses are assigned lower baseline confidence (0.60–0.75) due to higher ambiguity in general text.

---

How confidence scores are generated

The DLP pipeline produces confidence scores in a multi-stage process:

Stage 1: Regex baseline

Each DLP rule carries a confidence_threshold and an initial match confidence. Pattern rules define their own initial score in config_json.confidence_threshold (range 0.0–1.0). Built-in pattern rules use:

Entity type	Baseline confidence	Rationale
API keys, secrets, bearer tokens	0.95	High-entropy — very low false-positive rate
SSN, credit card, IBAN, passport	0.80	Structured but context-dependent
Phone numbers, email addresses	0.65	Common in legitimate text
Medical record numbers, NPI	0.70	Structured but common in clinical contexts

Custom rules created via POST /api/v1/admin/dlp-rules inherit the confidence_threshold you specify in the request body.

Stage 2: Context keyword boost

After regex detection, the pipeline runs the context keyword checker (between regex tier and DeBERTa validation). For each match, the engine scans a configurable character window (default: ±50 characters) around the match position for entity-specific reinforcing keywords.

If any boost keyword is found in the window, the match confidence is increased by the configured boost amount. The confidence is never decreased — context keywords are boost-only.

Key behavior:

• Only the first keyword hit applies — multiple keyword matches do not compound
• The boost is clamped to 1.0 (confidence cannot exceed 1.0)
• Matches with no keyword config are unchanged

Example boosts by entity type:

Entity type	Boost keywords (examples)	Boost amount
ssn	social security, ssn, tax id, itin	+0.15
credit_card	card, credit, debit, visa, payment	+0.15
passport_number, passport_us	passport, travel document, passport no	+0.20
drivers_license	driver, license, dl, driving	+0.15
phone_number, phone	phone, mobile, cell, tel, contact	+0.10
email	email, e-mail, send to, reply to	+0.05
iban	iban, bank, account, wire transfer	+0.10
aba_routing	routing, aba, routing number, transit	+0.15
swift_bic	swift, bic, correspondent, international	+0.15
npi	npi, provider, national provider, physician	+0.15
medical_record_number	mrn, medical record, patient, chart	+0.10
dea_number	dea, drug enforcement, controlled substance	+0.10
ein	ein, employer, tax id, federal	+0.10

Practical example:

A regex match for a pattern that looks like an SSN starts with confidence 0.80. The surrounding text contains "Social Security Number:". The context keyword "social security" is found within 50 characters of the match, boosting the score by +0.15 → final confidence 0.95. This pushes the match above many org confidence thresholds and makes it more likely to trigger a DeBERTa validation pass.

Stage 3: Co-occurrence boost

After context keyword checking, the pipeline runs the co-occurrence booster. This stage increases confidence for matches that appear alongside corroborating entity types in the same text window.

The intuition: if a text contains both an SSN and an email address near each other, the SSN match is more likely to be genuine than if the SSN appeared alone.

Key behavior:

• Boost-only — co-occurrence never decreases confidence
• Only one boost per entity type counted (not per match instance)
• Boost is proportional to the count of distinct nearby entity types, capped at a maximum
• Proximity is measured in characters between the end of one match span and the start of another

Co-occurrence rules:

Entity type	Corroborating entity types	Proximity window	Boost/neighbor	Max boost
ssn	email, phone_number, itin, drivers_license	200 chars	+0.05	+0.15
credit_card	email, phone_number, ssn	200 chars	+0.05	+0.15
passport_number	email, phone_number, ssn, drivers_license	200 chars	+0.05	+0.20
drivers_license	ssn, email, phone_number	200 chars	+0.05	+0.15
phone_number, phone	email, ssn	200 chars	+0.05	+0.10
email	ssn, phone_number, credit_card	200 chars	+0.03	+0.10
npi	dea_number, medical_record_number, ndc_code	200 chars	+0.05	+0.15
aba_routing	iban, swift_bic, credit_card, ein	200 chars	+0.05	+0.15
swift_bic	iban, aba_routing, credit_card	200 chars	+0.05	+0.15

Practical example:

Text contains: an email address, a phone number, and an SSN within a 200-character block (like a contact form dump). The SSN starts at confidence 0.80. Context keyword boost adds +0.15 → 0.95. Co-occurrence: 2 nearby entity types (email, phone) at +0.05 each → +0.10. Final SSN confidence: min(1.0, 0.95 + 0.10) = 1.0. The match is now at maximum confidence.

Stage 4: DeBERTa contextual validation (L3)

At Stage 4, high-confidence matches (above the L3 dispatch threshold) are routed to the DeBERTa contextual validator (when deployed). DeBERTa is the only stage that can decrease confidence — it may assign a very low score to a match that looks like PII syntactically but is not in context (e.g., "The SSN format is: 123-45-6789" in a documentation excerpt).

For L3 routing, the DeBERTa service returns a softmax probability (0–1) for the “sensitive content” class. This score is then routed through the L3 confidence threshold policy:

L3 confidence	Result
> 0.70	Hard block — regardless of org sensitivity setting
0.35–0.70 (dlp_sensitivity=high)	Soft block with elevated audit flag
0.35–0.70 (dlp_sensitivity=standard)	Pass with elevated audit flag
< 0.35	Pass — confidence falls below any reasonable pipeline threshold

The DeBERTa service is gated behind the DEBERTA_MICROSERVICE_URL configuration key. If the service is unavailable, the pipeline fails open (the match proceeds without L3 validation).

---

Per-rule confidence threshold

Every DLP rule has a confidence_threshold (0.0–1.0) that acts as a minimum filter. Matches whose final boosted confidence falls below this threshold are excluded from the scan result — they are detected but silently discarded.

Default thresholds by rule type:

Rule type	Default threshold
Secrets (API keys, tokens, bearer tokens)	0.95
Structured PII (SSN, credit card, passport)	0.80
Semi-structured (phone, email)	0.65
Custom regex rules	As specified by the admin

How to view and update thresholds:

# View current threshold for a rule
curl "https://gateway.arbitex.ai/api/v1/admin/dlp-rules/{rule_id}" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Update the confidence threshold
curl -X PUT "https://gateway.arbitex.ai/api/v1/admin/dlp-rules/{rule_id}" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"confidence_threshold": 0.85}'

Tuning guidance:

• Raise threshold (e.g., 0.65 → 0.85) — reduces false positives, increases false negatives. Use when a rule produces too many noisy alerts.
• Lower threshold (e.g., 0.80 → 0.65) — increases recall, may increase false positives. Use when you are missing genuine matches.
• The confidence_threshold is evaluated after all boosts are applied — boosting a match above the threshold is the intended mechanism for increasing effective recall on context-rich matches.

---

Admin-tunable settings

DLP_CONTEXT_WINDOW_CHARS

Controls the character window size for context keyword scanning. The checker scans this many characters before and after each regex match looking for boost keywords.

Setting	Default	Range	Location
DLP_CONTEXT_WINDOW_CHARS	50	10–200	System config (DB or env var)

Via admin API:

curl -X PUT "https://gateway.arbitex.ai/api/v1/admin/config/dlp_context_window_chars" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": "100"}'

Effect: Larger windows increase the chance of finding boost keywords at the cost of slightly higher CPU usage per scan. For typical conversational text (1–2 sentence context), 50 characters is adequate. For structured forms or dense documents with headers far from data values, increase to 100–150.

dlp_sensitivity org setting

The dlp_sensitivity org-level setting controls routing for ambiguous DeBERTa confidence scores (the 0.35–0.70 band). This is an org-level property, not a system config key.

Value	Ambiguous-band routing
standard (default)	Pass with elevated audit flag — alert only
high	Soft block with elevated audit flag

Update via org settings:

curl -X PUT "https://gateway.arbitex.ai/api/v1/admin/orgs/{org_id}/settings" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dlp_sensitivity": "high"}'

This setting also governs CredInt routing — "high" sensitivity soft-blocks on "critical" and "high" frequency bucket hits, while "standard" treats them as elevated-audit passes.

Per-rule confidence_threshold

Discussed in the section above. Configurable per rule via PUT /api/v1/admin/dlp-rules/{rule_id}.

---

Confidence flow summary

The complete confidence flow for a single match from detection to enforcement:

Regex pattern match
  → Initial confidence score (e.g., 0.80 for SSN pattern)

Context keyword check (±50 chars window)
  → Keyword found: confidence += boost_amount (e.g., +0.15)
  → No keyword: confidence unchanged
  → Max: 1.0

Co-occurrence check (±200 chars)
  → 1 corroborating type: confidence += boost_per_neighbor (e.g., +0.05)
  → 2+ types: up to max_boost (e.g., max +0.15)
  → No neighbors: unchanged

Confidence threshold filter
  → score < rule.confidence_threshold → match discarded
  → score ≥ threshold → match passes to policy evaluation

DeBERTa L3 (if enabled and match score is in L3 dispatch range)
  → L3 score > 0.70 → hard_block
  → L3 score 0.35–0.70 + dlp_sensitivity=high → soft_block
  → L3 score 0.35–0.70 + dlp_sensitivity=standard → pass (elevated audit)
  → L3 score < 0.35 → match confidence demoted below threshold → discard

---

Interpreting confidence in audit events

Confidence information appears in the DLP match data in audit events. Each matched entity in the extra_metadata.detected_entities field includes the final post-boost confidence score:

{
  "extra_metadata": {
    "detected_entities": [
      {
        "entity_type": "ssn",
        "confidence": 0.95,
        "detector": "regex_spacy",
        "action": "REDACT"
      }
    ],
    "action_taken": "REDACT",
    "match_reason": "entity_type matched ['ssn']"
  }
}

The confidence value in the audit event is the final post-boost score after all three calibration stages have run. This is the score that was compared against the rule’s confidence_threshold.

---

Related documentation

• DLP Pipeline Configuration
• DLP Detection Hierarchy
• DLP Accuracy Reference
• DeBERTa Tier 3 Admin
• DLP Rule Administration