Accuracy & Confidence Disclosure

Accuracy & Confidence Disclosure

Arbitex is committed to transparency about detection accuracy. This guide explains how we measure precision across entity types, how the multi-tier DLP pipeline achieves its accuracy targets, and how confidence thresholds map to enforcement actions.

---

Detection Accuracy Overview

Structured vs Context-Dependent Formats

Format Category	Precision Target	Examples	Notes
Structured (deterministic patterns)	≥ 0.95	AWS keys (AKIA prefix), Stripe keys (sk_live_ prefix), GitHub PATs (ghp_ prefix)	Fixed prefixes and checksums make regex highly reliable
Semi-structured	≥ 0.90	Connection strings, pre-signed URLs, webhook URLs	Recognizable structure but variable format
Context-dependent	Varies (0.70–0.95)	RSA/EC private keys, JWT signing keys, generic API tokens	Requires NLP context analysis; accuracy depends on surrounding text

Note

Precision = true positives / (true positives + false positives). A precision of 0.95 means at most 5% of detections are false positives.

---

Multi-Tier Detection Pipeline

The Arbitex DLP engine processes each piece of content through a 5-tier pipeline. Each tier narrows the candidate set and increases confidence before enforcement actions are taken.

Tier 1 — Regex Pattern Matching

• First pass: fast regex patterns scan content for candidate matches
• Each of the 39 entity types has one or more regex patterns
• High recall (catches nearly all real secrets) but moderate precision alone
• Example: /AKIA[0-9A-Z]{16}/ for AWS access keys — extremely precise due to fixed prefix

Tier 2 — NER (Named Entity Recognition)

• ML-based entity recognition validates regex candidates
• Reduces false positives from strings that match patterns but are not secrets
• Especially important for context-dependent types (generic tokens, private keys)

Tier 3 — DeBERTa NLI Validation

• Natural Language Inference model evaluates the context around each candidate
• Classifies context as: “this is a real credential” vs “this is example/documentation/test data”
• Dramatically reduces false positives in documentation, code comments, and test fixtures
• Context window: analyzes the surrounding text (keyword checking + co-occurrence boosting)

Tier 4 — CredInt (Credential Intelligence)

• Optional live validation against provider APIs (e.g., test if an AWS key is active)
• Available for select entity types (cloud provider keys, communication tokens)
• Provides definitive true/false but requires network access
• Disabled by default in air-gap deployments

Tip

Each tier narrows the candidate set. A detection that passes all 4 tiers has extremely high confidence. The admin dashboard shows which tiers contributed to each detection.

---

Confidence Scores & Thresholds

How Confidence is Calculated

• Each detection receives a confidence score from 0.0 to 1.0
• Score combines: regex match strength, NER probability, DeBERTa entailment score, CredInt verification (when available)
• Formula: weighted geometric mean of tier scores, with CredInt as a binary multiplier

Confidence Thresholds by Action Severity

Action	Minimum Confidence	Rationale
AUDIT (log only)	0.30	Low bar — capture everything for review, accept more false positives
PROMPT (user confirmation)	0.60	Moderate confidence — ask the user before allowing
ALLOW_WITH_OVERRIDE	0.60	Same as PROMPT — enforcement but override available
BLOCK (hard block)	0.80	High bar — must be confident to block content

Caution

These are the platform defaults. Administrators can adjust thresholds per entity type via the policy pack configuration. Lowering BLOCK thresholds increases security but may increase user friction from false positives.

Confidence Score Breakdown in Scan Results

Each scan result includes:

• confidence: overall score (0.0–1.0)
• tier_scores.regex: pattern match score
• tier_scores.ner: NER classification probability
• tier_scores.nli: DeBERTa entailment score
• tier_scores.credint: CredInt verification result (null if not run)

---

Golden Test Corpus Methodology

What is the Golden Test Corpus?

• Curated dataset of ~10,000 labeled samples per entity type
• Contains both positive (real secrets) and negative (look-alikes, examples, test data) samples
• Sourced from: synthetic generation, sanitized real-world samples, adversarial examples

Evaluation Methodology

Isolated Evaluation

• Each entity type evaluated independently against its golden test subset
• Metrics: precision, recall, F1, false positive rate, false negative rate
• Published per-entity metrics are from isolated evaluation

Integrated Evaluation

• Full pipeline run against mixed corpus (all 39 types intermixed)
• Tests for cross-type confusion (e.g., AWS key regex matching a Stripe key pattern)
• Evaluates pipeline throughput and latency under realistic load

CI Regression Gates

• Every model update (DeBERTa, NER, regex patterns) triggers a full golden test run
• Hard gates:
  • Precision must not drop below entity type’s published minimum
  • Recall must not drop more than 2% from previous release
  • No new cross-type confusion errors
• Soft gates (warnings, not blockers):
  • Latency increase > 10%
  • Confidence score distribution shift (Kolmogorov-Smirnov test)

---

Context Window Analysis

Keyword Checking

• For context-dependent types, the pipeline scans surrounding text for keywords
• Keywords include: “password”, “secret”, “token”, “key”, “credential”, “auth”, “bearer”
• Presence of keywords boosts confidence; absence reduces it
• Negative keywords reduce confidence: “example”, “test”, “dummy”, “placeholder”, “fake”, “sample”, “xxx”

Co-occurrence Boosting

• When multiple credential-related artifacts appear near each other, confidence increases
• Example: a string matching an AWS access key pattern near a string matching an AWS secret key pattern — both get a co-occurrence boost
• Co-occurrence radius: within 500 characters or 10 lines
• Boost magnitude: +0.10 to +0.15 on the overall confidence score

---

Per-Entity Accuracy Metrics

Accessing Metrics

• Admin UI: Settings → Entity Types → click any type → Accuracy tab
• API: GET /v1/entity-types/{type}/accuracy — returns precision, recall, F1, FP rate, sample size, last evaluation date

Interpreting the Metrics

Entity Type	Precision	Recall	FP Rate	Format Category
aws_access_key	0.99	0.98	0.01	Structured
github_pat	0.98	0.97	0.02	Structured
stripe_secret_key	0.99	0.99	0.01	Structured
postgres_connection_string	0.94	0.91	0.06	Semi-structured
slack_webhook_url	0.96	0.95	0.04	Semi-structured
jwt_signing_key	0.85	0.82	0.15	Context-dependent
private_key_rsa	0.88	0.90	0.12	Context-dependent
private_key_ec	0.86	0.88	0.14	Context-dependent

Note

These metrics are from the most recent golden test evaluation. Actual production accuracy may vary based on your content mix. The accuracy API provides live metrics based on your organization’s detection history.

---

Improving Accuracy for Your Organization

Reporting False Positives

1. In the scan results, click Report False Positive on any detection
2. Provide context (why this is a false positive)
3. The report feeds into the golden test corpus for future model training
4. Immediate effect: the specific string is added to your org’s allowlist

Tuning Confidence Thresholds

• Increase thresholds for entity types with high false positive rates in your content
• Decrease thresholds for entity types where you want maximum security coverage
• Use AUDIT mode to test threshold changes before enforcing

Custom Allowlists

• Add known-safe patterns to your org’s allowlist (Settings → Allowlists)
• Supports exact string match and regex patterns
• Allowlisted patterns bypass DLP entirely — use with caution

---

Related Resources

• Entity Type Split & Policy Packs — the 39 entity types and policy pack system
• API Reference — Batch 39 — accuracy metrics API and entity enumeration API