DeBERTa Tier 3 — admin guide

Tier 3 adds contextual classification to the Outpost DLP pipeline. It runs a fine-tuned DeBERTa ONNX model directly inside the Outpost process and classifies each text chunk as pii or clean. Chunks classified as pii with confidence ≥ 0.70 are appended to the merged findings from Tier 1 (regex) and Tier 2 (NER) and passed to the Policy Engine for enforcement.

The current production model is deberta-dlp-v4. A future model version (v6) is planned but not yet released; see Roadmap below.

Tier 3 is optional. When DEBERTA_MODEL_PATH is not set or the ONNX file is not present, the pipeline runs Tier 1 + Tier 2 only and logs an INFO message. No configuration change is required to run without Tier 3.

---

What Tier 3 does

Tier 1 and Tier 2 identify entity spans using pattern matching and named-entity recognition. Tier 3 provides a second-pass contextual validation layer:

1. The full request or response text is split into overlapping chunks of up to 450 characters at sentence boundaries.
2. Each chunk is tokenized (max 512 tokens) and passed through the DeBERTa model.
3. The model outputs logits for two classes: pii (class 0) and clean (class 1).
4. The softmax probability of pii is compared against the 0.70 confidence threshold.
5. Chunks that exceed the threshold produce a finding with tier: "deberta" and action_tier: "redact".

The model runs entirely within the Outpost — it makes no calls to Platform or Cloud.

Model version — deberta-dlp-v4 (production)

Model: deberta-dlp-v4 (DeBERTa-v3-large, 304M parameters). Training corpus: 2.8M examples (v3 base + loghub + code_secrets + id_confusion + medical + chat). Trained on 8x H100 SXM.

Eval accuracy (v4)

Eval split: 281K examples across 35 entity types. Accuracy: 99.48%.

23 entity types achieve perfect F1 (1.000). 12 entity types achieve strong F1 (0.984–0.998). See DLP Accuracy Reference for the full per-entity breakdown.

Adversarial accuracy (v4)

Adversarial expanded set: 2,800 examples across 10 categories. Accuracy: 98.32%. Precision: 0.986, Recall: 0.981, F1: 0.984.

Category	Accuracy	What it tests
code_fixture_fp	100%	Test fixtures, mock data, example code
credential_env_tp	100%	Real API keys, connection strings, private keys
fictional_fp	100%	Simpsons addresses, example.com, synthetic data
format_obfuscation_tp	100%	Spaced/dashed/dotted real PII
infrastructure_ip_fp	100%	Log lines, load balancer IPs, RFC 1918
public_mnpi_fp	100%	Published earnings, announced M&A
structured_data_tp	100%	CSVs, JSON, tabular real PII
mnpi_coded_tp	98.3%	Coded/euphemistic MNPI references
redacted_fp	94.3%	Redacted/masked data
ai_framing_tp	93.0%	Jailbreak-framed real PII

The three sub-100% categories reflect genuinely hard classification boundaries: coded MNPI uses euphemistic language that can be ambiguous even for human reviewers; redacted data occasionally retains enough structure to look real; and jailbreak-framed PII deliberately wraps real data in adversarial prompt structures. A future model version targets these three gaps.

V2 to v4 improvements

The following entity types had significant false positive issues under v2 adversarial conditions. V4 resolves all of them:

Entity	V2 adversarial	V4
indian_aadhaar	100% FP rate	F1 = 1.000
ein	75% FP rate	F1 = 1.000
npi	73% FP rate	F1 = 1.000
uk_nhs_number	76% FP rate	F1 = 1.000
uk_nino	100% FP rate	F1 = 0.984

Over-hinting validation

A/B testing confirmed that NLI hypothesis qualifiers (“real”, “specific person”) have zero effect on classification accuracy for PII entity types. The model learns the TP/FP distinction from text context, not from hypothesis hints. The only qualifier that affects accuracy is the MNPI qualifier (“not yet publicly announced”) — this is by design, since the temporal dimension of MNPI materiality cannot be inferred from text context alone.

---

Upgrading the model

When a future model version (v6 or later) is released and passes all accuracy and adversarial gates:

1. Download the new ONNX artifact to the model directory.
2. Update DEBERTA_MODEL_PATH to point to the new model.onnx.
3. Restart the Outpost. The startup log will confirm the loaded model.
4. No configuration changes are required — future versions use the same label order (class 0 = pii, class 1 = clean), the same confidence threshold (0.70), and the same tokenizer family.

The v4 model can be kept as a rollback artifact. To revert, point DEBERTA_MODEL_PATH back to the v4 model.onnx and restart.

---

Performance characteristics

Metric	Value
Average inference latency (CPU)	88.9 ms per request
P95 inference latency (CPU)	164 ms per request
Chunk size	450 chars / 512 tokens max
Minimum confidence for reporting	0.70

CPU is functional but GPU is recommended for production. On CPU, each inference call adds ~89 ms to the DLP pipeline. On a GPU node (NVIDIA T4 or equivalent), inference drops to 5–15 ms per chunk.

---

Deployment

Model artifact

The promoted ONNX artifact for the current production model (deberta-dlp-v4) is:

deberta-dlp-v4/
├── model.onnx          ← required
├── tokenizer.json      ← required
├── tokenizer_config.json
└── vocab files         ← required by AutoTokenizer

DEBERTA_MODEL_PATH must point to model.onnx. The tokenizer is loaded from the same directory (os.path.dirname(DEBERTA_MODEL_PATH)).

Bare-metal or Docker deployment

Set the environment variable to the model file path:

DEBERTA_MODEL_PATH=/opt/arbitex/models/deberta/model.onnx
DLP_DEBERTA_ENABLED=true

Mount the model directory into the container:

docker-compose.yml

volumes:
  - /host/path/to/deberta-dlp-v2:/app/models/deberta:ro
environment:
  DEBERTA_MODEL_PATH: /app/models/deberta/model.onnx
  DLP_DEBERTA_ENABLED: "true"

Kubernetes / Helm deployment

Add a model volume to the Outpost pod and set the Helm values:

values-override.yaml

outpost:
  dlpDebertaEnabled: true
  debertaModelPath: /app/models/deberta/model.onnx
  gpuEnabled: true   # set false for CPU-only deployment

# Add a volume for the model
extraVolumes:
  - name: deberta-model
    persistentVolumeClaim:
      claimName: deberta-model-pvc   # or hostPath, NFS, etc.

extraVolumeMounts:
  - name: deberta-model
    mountPath: /app/models/deberta
    readOnly: true

Deploy:

helm upgrade arbitex-outpost ./charts/arbitex-outpost \
  -f values-override.yaml \
  --set outpost.dlpDebertaEnabled=true \
  --set outpost.debertaModelPath=/app/models/deberta/model.onnx

For GPU nodes, the chart sets resource requests for nvidia.com/gpu: 1 when outpost.gpuEnabled: true.

Python dependencies

The Outpost image must include at least one of these dependency sets:

Runtime	Required packages	Notes
Preferred (optimum ORT)	transformers, optimum[onnxruntime], torch	Richer HuggingFace API
Fallback (raw onnxruntime)	transformers, onnxruntime	Minimal deps, CPU only

Install:

# Preferred
pip install transformers optimum[onnxruntime] torch

# Minimal fallback
pip install transformers onnxruntime

---

Configuration reference

All settings are environment variables. The Outpost image uses Pydantic settings — prefix-free, case-insensitive.

Environment variable	Helm value	Type	Default	Description
DLP_DEBERTA_ENABLED	outpost.dlpDebertaEnabled	bool	false	Enable Tier 3. Must also set DEBERTA_MODEL_PATH.
DEBERTA_MODEL_PATH	outpost.debertaModelPath	string	""	Absolute path to model.onnx. When empty, Tier 3 is inactive.
GPU_ENABLED	outpost.gpuEnabled	bool	false	Request GPU resources (nvidia.com/gpu: 1). Enables CUDA execution provider.
DLP_ENABLED	outpost.dlpEnabled	bool	true	Master DLP toggle. Tier 3 requires this to be true.

Confidence threshold

The confidence threshold is fixed at 0.70 in the current model version. Chunks where P(pii) < 0.70 are silently discarded. Chunks at or above 0.70 generate a finding with:

{
  "entity_type": "pii",
  "tier": "deberta",
  "action_tier": "redact",
  "confidence": 0.84
}

The Policy Engine then applies compliance bundle rules. If a bundle’s DLP action for pii is log_only or block, that takes precedence over the Tier 3 default redact.

Escalation band

Tier 3 outputs only pii or clean — it does not identify specific entity types (SSN, email, etc.). Entity-type specificity comes from Tier 1 and Tier 2. Tier 3 confirms or discards their findings contextually.

---

Monitoring

Startup log

On successful load, the Outpost logs at INFO level:

INFO     outpost.dlp.deberta  DeBERTa Tier 3 loaded via optimum ORT — contextual classification active

or:

INFO     outpost.dlp.deberta  DeBERTa Tier 3 loaded via onnxruntime — contextual classification active

If the model is not configured:

INFO     outpost.dlp.deberta  DeBERTa Tier 3 not configured — using regex+NER only. Set DEBERTA_MODEL_PATH to enable.

Audit log entries

Each Tier 3 finding writes an audit log entry with the following fields:

Field	Value
tier	deberta
action_tier	redact (default, may be overridden by Policy Engine)
confidence	Float, e.g. 0.847
entity_type	pii

To extract Tier 3 audit events:

# On the Outpost host
jq 'select(.tier == "deberta")' audit_buffer/audit.jsonl

# Via Platform audit API
GET /api/v1/admin/audit?tier=deberta&limit=100

Confirm vs discard

• Confirm: P(pii) ≥ 0.70 — chunk appended to findings, forwarded to Policy Engine.
• Discard: P(pii) < 0.70 — chunk ignored, no audit entry written for the discarded chunk.

The ratio of confirmed to discarded chunks appears in debug logs when LOG_LEVEL=debug.

---

Troubleshooting

Tier 3 not activating

Symptom: Startup log shows “DeBERTa Tier 3 not configured” even after setting DEBERTA_MODEL_PATH.

Check:

1. Confirm the file exists at the configured path inside the container:

   kubectl exec <pod> -- ls -la /app/models/deberta/model.onnx

2. Confirm DLP_DEBERTA_ENABLED=true is set.
3. Confirm the volume mount is correct — the path must match DEBERTA_MODEL_PATH exactly, pointing to the .onnx file, not the directory.

Model load failure

Symptom: Warning log: Failed to load DeBERTa ONNX model from '...': <error>. Falling back to regex+NER only.

Common causes and fixes:

Error message	Cause	Fix
transformers not installed	Package missing	pip install transformers onnxruntime
No such file or directory	Wrong path	Verify mount and DEBERTA_MODEL_PATH value
ORT ONNX model load failed	Corrupt or incompatible ONNX file	Re-export model with matching ONNX opset
Cannot allocate memory	Insufficient container memory	Increase memory limit (minimum 2 Gi for CPU, 8 Gi for GPU)

Incorrect label mapping

Symptom: Tier 3 consistently classifies clean text as pii or vice versa.

Root cause: deberta-dlp-v4 uses an inverted label order relative to the original spec — class 0 = pii, class 1 = clean. The runtime in outpost/dlp/deberta.py accounts for this correctly. If you load the model with a custom inference script, ensure DEBERTA_ENTITY_LABELS[0] = "pii" and DEBERTA_ENTITY_LABELS[1] = "clean".

Do not use argmin(logits) — use argmax(softmax(logits)) with the ["pii", "clean"] label order.

High latency on CPU

Symptom: Each DLP scan adds 100–200 ms to request latency.

Fix: Move the Outpost to a GPU node and set GPU_ENABLED=true. GPU inference reduces latency to 5–15 ms per chunk. Alternatively, reduce the proportion of long-text requests that require chunking, or disable Tier 3 for latency-sensitive Policy Pack paths using dlp_deberta_enabled=false in the relevant compliance bundle.

ONNX Runtime version mismatch

Symptom: InvalidGraph: Load model from ... failed: ... opset X not supported.

Fix: The model was exported with a specific ONNX opset. Install the matching onnxruntime version:

pip install onnxruntime==1.17.3   # or the version used during export

Check the model’s opset:

import onnx
m = onnx.load("/app/models/deberta/model.onnx")
print(m.opset_import)

---

Roadmap

v6 planned. A future model version (v6) is planned to improve contextual accuracy on the hardest adversarial categories (redacted false positives and coded MNPI). v6 will use the same DeBERTa-v3-large architecture and ONNX export pipeline, so no customer configuration changes are required when it ships. Release timeline is not yet committed. v4 remains the production model until a future version passes all adversarial gates.