ScanEngine Architecture

The ScanEngine is the unified content extraction and DLP scanning library in arbitex-core. All Arbitex channels — AI Gateway, Email Relay, File Inspector, and the Hybrid Outpost — import and call ScanEngine in-process. No new network hops are added to the hot path. The legacy DLPPipeline class is deprecated — ScanEngine is the sole orchestrator for all DLP scanning.

Phase 3 capabilities

ScanEngine Phase 3 adds:

Early termination — A BLOCK action at any tier short-circuits the remaining pipeline. ScanEngine checks the resolved action after each tier; if BLOCK, remaining tiers are skipped and the result is returned immediately.
Per-tier OTel metrics — Each tier emits OpenTelemetry counters, histograms, and gauges for observability. See DLP Pipeline Architecture — OTel Metrics for the full metric catalog.
Entity normalization — All detector output is normalized to a consistent DLPMatch schema before deduplication, regardless of detector backend.
Async CredInt — Tier 4 (CredInt) runs as a concurrent asyncio.create_task alongside the Tier 1→2→3 chain, so its latency does not add to the serial scan path.

Architecture

ScanEngine composes two subsystems: the extraction registry (format-specific text extraction) and the DLP pipeline (sensitive data detection). Content flows through extraction first, then DLP:

flowchart TD
    IN([Raw content\nbytes + MIME type]) --> SE[ScanEngine\norchestrator]

    SE --> DECIDE{Text-like\nMIME type?}
    DECIDE -->|Yes| DIRECT[Direct UTF-8 decode\nno extraction overhead]
    DECIDE -->|No| REG[ExtractorRegistry\nlazy MIME → handler lookup]

    REG --> PDF[PdfExtractor\npdfplumber]
    REG --> DOCX[DocxExtractor\npython-docx]
    REG --> XLSX[XlsxExtractor\nopenpyxl]
    REG --> PPTX[PptxExtractor\npython-pptx]
    REG --> IMG[ImageExtractor\nOCR backend]
    REG --> OTHER[CSV / RTF / TXT\nstdlib + striprtf]

    PDF & DOCX & XLSX & PPTX & IMG & OTHER --> ET([ExtractedText\ntext + method + metadata])
    DIRECT --> ET

    ET --> T0{Tier 0\nTF-IDF pre-filter\ninput only}
    T0 -->|blocked| BLOCK0([BLOCK — early termination\nskip remaining tiers])
    T0 -->|pass / disabled| DLP

    subgraph DLP["DLP Pipeline (ScanEngine orchestration)"]
        direction TB
        T1[Tier 1 — Regex] --> T1_CHECK{BLOCK?}
        T1_CHECK -->|Yes| BLOCK1([Early termination])
        T1_CHECK -->|No| T2[Tier 2 — NER]
        T2 --> T3[Tier 3 — DeBERTa v4/v5]
        CREDINT[Tier 4 — CredInt\nasync concurrent] -.->|parallel| T1
    end

    BLOCK1 --> SR
    T3 --> SR([ScanResult\nextraction metadata + DLP matches\n+ OTel metrics + action])
    CREDINT --> SR
    BLOCK0 --> SR

Key design principles

Library, not microservice. Zero new network hops. Current hop pattern preserved for every consumer.
Flat registries. MIME types map to extractors via a flat dict. Detectors are a flat list. No class hierarchies.
Lazy imports. pdfplumber is only imported when a PDF arrives. paddleocr is only imported when an image arrives. Dependencies are optional extras.
Single responsibility per file. Each format handler is one file implementing ExtractorProtocol. Each DLP component is a standalone module.
No cross-knowledge. No extractor knows about any other extractor. No detector knows about any other detector.

Supported Formats

The extraction registry supports these MIME types out of the box:

Format	MIME Type	Extractor	Library	Install Extra
PDF	`application/pdf`	`PdfExtractor`	pdfplumber	`arbitex-core[pdf]`
DOCX	`application/vnd.openxmlformats-officedocument.wordprocessingml.document`	`DocxExtractor`	python-docx	`arbitex-core[docx]`
XLSX	`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`	`XlsxExtractor`	openpyxl	`arbitex-core[xlsx]`
PPTX	`application/vnd.openxmlformats-officedocument.presentationml.presentation`	`PptxExtractor`	python-pptx	`arbitex-core[pptx]`
RTF	`text/rtf`, `application/rtf`	`RtfExtractor`	striprtf	`arbitex-core[rtf]`
CSV	`text/csv`	`CsvExtractor`	stdlib	(included)
Plain text	`text/plain`	`TxtExtractor`	stdlib	(included)
Images	`image/png`, `image/jpeg`, `image/tiff`, `image/bmp`, `image/webp`	`ImageExtractor`	PaddleOCR or Tesseract	`arbitex-core[ocr-paddle]` or `arbitex-core[ocr-tesseract]`

Text-like MIME types (text/plain, text/csv, application/json, application/xml, text/html, text/markdown) bypass the extraction registry entirely — content is decoded as UTF-8 directly.

Install all document extractors at once:

pip install arbitex-core[extraction]
# Includes: pdf, docx, xlsx, pptx, rtf

OCR Backend Selection

Image extraction delegates to a pluggable OCR backend selected by the OCR_BACKEND environment variable:

Deployment	Backend	Value	Hardware	Air-Gap Safe
SaaS (platform)	PaddleOCR (default)	`paddleocr`	Existing GPU service	Yes
Outpost (with GPU)	PaddleOCR	`paddleocr`	Customer GPU	Yes
Outpost (CPU-only)	Tesseract (fallback)	`tesseract`	No GPU required	Yes

PaddleOCR is the default because it provides CJK/multilingual support (80+ languages), handwriting recognition, and better accuracy on real-world documents (rotated, skewed, low-contrast). Models are baked into the Docker image at build time — never fetched at runtime.

Tesseract is available as a CPU-only fallback for outpost deployments without GPU hardware.

Both backends implement the OCRBackend protocol and are interchangeable. The ImageExtractor selects the backend at initialization and reports the backend name in the extraction_method field (e.g., image/paddleocr or image/tesseract).

Configuration

Environment variables

Variable	Default	Description
`OCR_BACKEND`	`paddleocr`	OCR backend: `paddleocr` (GPU) or `tesseract` (CPU)
`DLP_SCANNER_BACKEND`	`python`	Tier 1 regex backend: `python` (re module) or `rust` (compiled scanner via PyO3)

Optional dependency extras

# Individual format support
pip install arbitex-core[pdf]
pip install arbitex-core[docx]
pip install arbitex-core[xlsx]
pip install arbitex-core[pptx]
pip install arbitex-core[rtf]

# OCR backends
pip install arbitex-core[ocr-paddle]    # PaddleOCR + Pillow
pip install arbitex-core[ocr-tesseract] # pytesseract + Pillow

# All document formats (no OCR)
pip install arbitex-core[extraction]

Usage Patterns

Core ScanEngine (synchronous)

The ScanEngine class in arbitex_core.dlp.scan_engine is the primary interface for synchronous consumers:

from arbitex_core.dlp.scan_engine import ScanEngine

engine = ScanEngine(detectors=[regex_detector, ner_detector])

# Scan a file
result = engine.scan(file_bytes, "application/pdf", filename="report.pdf")

# Check result
if result.blocked:
    # DLP triggered BLOCK action
    handle_block(result.matches)
elif result.action == DLPAction.REDACT:
    # Use sanitized text
    safe_text = result.modified_text
else:
    # Clean or log-only
    process(result.extracted_text)

The ScanResult dataclass contains both extraction metadata and DLP pipeline output:

Field	Type	Description
`extracted_text`	`str`	Text extracted from content
`extraction_method`	`str`	How text was extracted (`pdf`, `docx`, `image/paddleocr`, `direct`, etc.)
`extraction_metadata`	`dict`	Format-specific metadata (page count, word count, etc.)
`extraction_failed`	`bool`	Whether extraction failed
`extraction_error`	`str \| None`	Error message when extraction fails
`matches`	`list[DLPMatch]`	DLP matches after pipeline processing
`action`	`DLPAction`	Highest-severity action: `LOG_ONLY`, `REDACT`, or `BLOCK`
`modified_text`	`str \| None`	Text after redaction (if `REDACT` action)
`blocked`	`bool`	Whether the request should be blocked
`detector_count`	`int`	Number of detectors that ran
`match_count`	`int`	Number of final matches
`matched_categories`	`list[str]`	Sorted unique entity types

Platform AsyncScanEngine (asynchronous)

The platform wraps the core library with an async interface in AsyncScanEngine:

from backend.app.core.scan_engine import AsyncScanEngine

engine = AsyncScanEngine(
    detectors=[(regex_detector, DLPAction.REDACT)],
    validator=deberta_validator,
    context_checker=context_keyword_checker,
    co_occurrence_booster=co_occurrence_booster,
)

# Scan pre-extracted text (chat prompts)
result = await engine.scan(prompt_text, direction="input")

# Scan binary content (files, email attachments)
result = await engine.scan_content(file_bytes, "application/pdf", filename="report.pdf")

The async wrapper adds platform-specific features:

Concurrent detector execution via asyncio.gather
DeBERTa NLI validation as a second-pass contextual validator
Context keyword boosting for nearby keyword reinforcement
Co-occurrence boosting for corroborating entities
Per-entity confidence thresholds for precision calibration
Structural allowlist filtering (suppresses fictional values, flags test-mode keys)
Direction-aware action maps with separate input/output action defaults

Consumer integration

Consumer	Integration	Hop Count
AI Gateway (chat)	In-process `AsyncScanEngine.scan()` on text	0
File Upload	In-process `AsyncScanEngine.scan_content()`	0
File Inspector	In-process `ScanEngine.scan()`	0
Email Relay	HTTP to backend internal endpoint, then in-process scan	1 (unchanged)
Hybrid Outpost	Python sidecar imports `ScanEngine`	1 (unchanged)

DLP Pipeline Tiers

The full DLP pipeline is a 5-tier cascade. The core library implements Tiers 1–3 directly; Tier 0 and Tier 4 (CredInt) are implemented by the platform and outpost consumers.

Tier	Name	Technology	Purpose	Latency
0	TF-IDF pre-filter	sklearn TF-IDF + Logistic Regression	Prompt injection short-circuit	< 1 ms
1	Regex pattern matching	Compiled patterns (Python `re` or Rust via PyO3)	Structural secrets, PII, financial, medical	< 1 ms (p50)
2	NER entity extraction	Presidio, GLiNER, or GPU microservice	Person names, locations, MNPI	10–200 ms
3	DeBERTa contextual validation	NLI classifier — v4 production, v5 pending eval (microservice or local ONNX)	False positive demotion	20–100 ms
4	CredInt credential intelligence	Breach corpus (HTTP service or local bloom filter)	Known-compromised credential lookup	< 5 ms (concurrent)

Tier 0: TF-IDF prompt injection pre-filter

Tier 0 is a fast pre-filter that runs on input only, before the remaining tiers. It uses a serialized TF-IDF + logistic regression pipeline (tier0_model.joblib) to detect obvious prompt injection attempts. When the classifier’s confidence exceeds the threshold (default: 0.95), the request is blocked immediately and Tiers 1–4 are skipped entirely.

Model missing: fail-open (pass everything, tier disabled)
Classification failure: platform fails closed (blocks); outpost fails open (passes)
Output direction: Tier 0 does not run on output — only input prompts are classified

Configuration: see Tier 0 Pre-filter Guide.

Tiers 1–3: Core DLP pipeline

The core library’s DLP pipeline processes extracted text through these stages:

Detectors — Run all configured detectors (regex, NER, etc.) to produce raw matches.
Confidence filtering — Filter matches by per-entity confidence thresholds.
Deduplication — Remove overlapping spans, preferring higher confidence. Different entity types on the same span are kept (lower-priority entity gets 0.5x confidence penalty).
Structural allowlist — Suppress known-fictional values (NANPA 555-01xx, never-assigned SSNs) and flag test-mode keys.
Action resolution — Determine highest-severity action from remaining matches using entity-to-action maps.
Redaction — If action is REDACT, replace matched spans with [REDACTED] markers in reverse position order.

If the resolved action is BLOCK, a DLPBlockedError is raised (platform) or the result’s blocked flag is set (core).

Tier 4: CredInt (concurrent)

CredInt runs concurrently with the Tier 1–3 chain (via asyncio.create_task on the platform, or inline in the outpost). It checks credential candidates against a breach corpus using k-anonymity (only SHA-1 prefixes are transmitted). On the outpost, CredInt uses an on-disk bloom filter with zero network calls at scan time.

For a deep dive into per-tier detection capabilities, see the DLP Pipeline Architecture.

Deployment Matrix

Tier availability varies by deployment profile. The extraction registry is identical everywhere; the differences are in the DLP tiers.

Tier	SaaS (platform)	Outpost (full)	Outpost (minimal)
Tier 0 (TF-IDF)	Enabled by default	Enabled if `tier0_model.joblib` present	Disabled (no model file)
Tier 1 (Regex)	Python or Rust backend	Python or Rust backend	Python or Rust backend
Tier 2 (NER)	GPU microservice (`ner-gpu:8200`)	Local spaCy (`en_core_web_sm`)	Disabled (`DLP_NER_ENABLED=false`)
Tier 3 (DeBERTa)	Microservice (`deberta-validator:8201`)	Local ONNX model	Disabled (no model file)
Tier 4 (CredInt)	HTTP service (`CredentialIntelligenceClient`)	Local bloom filter (`.arbf`)	Disabled (no bloom file)
Compiled Rust scanner	Optional (`DLP_SCANNER_BACKEND=rust`)	Optional (`DLP_SCANNER_BACKEND=rust`)	Optional
OCR backend	PaddleOCR (GPU)	PaddleOCR (GPU) or Tesseract (CPU)	Tesseract (CPU)

Outpost minimal refers to an outpost deployment with only the base container and no GPU hardware, ONNX model, or bloom filter. In this mode, only Tiers 1 (regex) operates, providing structural pattern detection for secrets, PII, and financial data.

Fail modes differ by tier and deployment:

Platform microservice tiers (NER, DeBERTa): controlled by DLP_INFERENCE_FAIL_MODE — default closed (block on service failure)
Outpost local tiers: fail gracefully — a missing model file disables the tier at startup
Tier 0: platform fails closed on classification error; outpost fails open