Performance tuning guide

Arbitex Outpost runs a five-tier DLP pipeline — TF-IDF pre-filter, regex, NER, DeBERTa, and CredInt bloom filter checks. Each tier has a distinct latency and resource profile. This guide walks through every tuning lever available to operators: device selection, concurrency, tier skipping, benchmarking methodology, and container resource management. Before changing any setting in production, establish a documented baseline so you can measure regression or improvement objectively. See also: Outpost Operations, DLP Accuracy, DLP Detection Hierarchy, Deployment Topologies.

---

1. Performance baseline

Every tuning effort starts with a measured baseline. Without one, you cannot distinguish a real improvement from noise.

Default latency profile

The following figures are measured on a single-request, single-worker execution path on representative hardware (8-core x86 CPU, NVIDIA A10G GPU for the GPU column). Real-world numbers vary with payload size, entity density, and concurrent load.

Pipeline tier	CPU latency	GPU latency	Notes
Regex (Tier 1)	~1 ms	~1 ms	Regex is CPU-bound; GPU provides no benefit
NER (Tier 2)	~10–15 ms	~10–15 ms	Lightweight transformer; minimal GPU speedup at low batch size
DeBERTa (Tier 3)	~50–80 ms	~5–10 ms	5–10x GPU speedup; primary optimization target
CredInt bloom filter	~0.1 ms	~0.1 ms	In-memory bitset; negligible cost
Total (all tiers)	~60–100 ms	~10–20 ms	End-to-end scan of a typical 512-token request

Note

Payload size matters. DeBERTa processes sequences up to 512 tokens. Requests that exceed this are chunked, which multiplies per-chunk latency. Monitor outpost_scan_token_count histograms to understand your actual token distribution.

Establishing your baseline

Use the accuracy harness tool bundled with Outpost to capture a baseline before making any changes:

# Run harness in benchmark mode against a running Outpost instance
outpost-accuracy-harness \
  --target http://localhost:8080 \
  --corpus ./test-corpus/ \
  --benchmark-mode \
  --output baseline-$(date +%Y%m%d).json

The harness replays the test corpus through the live Outpost instance and records latency percentiles alongside accuracy metrics. Store the output file in version control alongside any configuration change that follows.

Tip

Run the baseline harness during representative load — not on an idle instance. Latency under concurrent pressure is what matters in production.

---

2. GPU vs CPU trade-offs

The choice between GPU and CPU execution is almost entirely determined by whether you have DeBERTa (Tier 3) enabled.

Where GPU helps

DeBERTa is the only tier with a meaningful GPU speedup:

• Tier 3 DeBERTa: 5–10x faster on GPU. A request that takes 60–80 ms CPU-side drops to 5–10 ms on a modern data-center GPU.
• Batch efficiency: GPU throughput scales with batch size. Under sustained load, the GPU amortizes inference cost across concurrent requests more efficiently than CPU threading.

Where GPU does not help

• Tier 1 regex: Pure CPU operation. GPU offload adds overhead without benefit.
• Tier 2 NER: The lightweight transformer used for NER is too small to saturate GPU compute. At the batch sizes typical for a single Outpost instance, CPU and GPU NER performance are comparable.

Configuring the inference device

Set the INFERENCE_DEVICE environment variable:

# GPU (recommended for production with DeBERTa enabled)
INFERENCE_DEVICE=cuda

# CPU (trial, dev, or DeBERTa-disabled deployments)
INFERENCE_DEVICE=cpu

Caution

Setting INFERENCE_DEVICE=cuda on a host without a CUDA-capable GPU will cause Outpost to fail at startup. Validate GPU availability before deploying GPU-mode containers.

Decision matrix

Deployment scenario	Recommended device
Production, DeBERTa enabled, latency SLA < 20 ms	cuda
Production, DeBERTa enabled, latency SLA < 100 ms	Either; benchmark to decide
Production, DeBERTa disabled (DEBERTA_ENABLED=false)	cpu
Trial, development, or CI	cpu
Edge/constrained hardware without GPU	cpu + ONNX runtime (see below)

CPU-only ONNX mode

For environments without GPU hardware, Outpost ships an ONNX-quantized DeBERTa model that reduces CPU inference time by approximately 30–40% compared to full FP32:

INFERENCE_DEVICE=cpu
DEBERTA_RUNTIME=onnx
DEBERTA_QUANTIZATION=int8

ONNX INT8 mode trades a small amount of accuracy for significantly reduced CPU latency and memory footprint. Review the accuracy trade-offs in DLP Accuracy before enabling in production.

---

3. Concurrency tuning

Outpost uses a fixed worker thread pool for inference. Concurrency configuration directly affects throughput and tail latency under load.

Worker threads

# Default: 4 worker threads
WORKER_THREADS=4

Increase WORKER_THREADS when:

• QPS is high and CPU utilization is below saturation
• NER tier is the throughput bottleneck (CPU-bound NER with many concurrent requests)
• You observe queue depth growing without CPU saturation (workers are I/O or lock-waiting)

Do not increase WORKER_THREADS when:

• Running GPU inference: GPU memory is shared across workers. Over-provisioning threads causes GPU memory contention and can trigger OOM errors.
• Running on a single-core host: thread overhead will exceed benefit.
• outpost_active_connections is near MAX_CONCURRENT_REQUESTS: increase the connection limit first.

A practical starting point: set WORKER_THREADS to the number of physical CPU cores minus 1 (leave one core for the OS and network stack).

# 8-core host example
WORKER_THREADS=7

Maximum concurrent requests

# Default: 50
MAX_CONCURRENT_REQUESTS=50

This cap prevents unbounded queue growth under burst traffic. When the limit is reached, Outpost returns HTTP 429 to the caller. Tune this in conjunction with WORKER_THREADS:

WORKER_THREADS	Suggested MAX_CONCURRENT_REQUESTS
4 (default)	50
8	100
16	200

Queue depth and backpressure

Outpost exposes outpost_request_queue_depth as a Prometheus gauge. Monitor this metric:

• Steady-state queue depth near 0: healthy. Workers are keeping up.
• Queue depth consistently > WORKER_THREADS × 2: workers are saturated. Increase WORKER_THREADS or scale horizontally.
• Queue depth spiking then draining: burst traffic. Consider increasing MAX_CONCURRENT_REQUESTS to absorb spikes, or deploy an upstream rate limiter.

Tip

Horizontal scaling (multiple Outpost replicas behind a load balancer) is almost always preferable to over-threading a single instance. See Deployment Topologies for multi-instance patterns.

---

4. DLP pipeline optimization

Not every request needs all five tiers. Skipping expensive tiers when they are not needed is the highest-leverage optimization available.

Tier skipping

Disable DeBERTa (Tier 3):

DEBERTA_ENABLED=false

With DeBERTa disabled, all requests stop at Tier 2 (NER). This reduces end-to-end latency from ~60–100 ms to ~11–16 ms on CPU. Appropriate when:

• Your policy only targets entity types reliably caught by regex + NER
• You are running in a latency-critical path and DeBERTa’s incremental accuracy gain is not required
• You are on CPU-only infrastructure and cannot meet your SLA with DeBERTa enabled

Disable CredInt bloom filter:

CREDINT_ENABLED=false

CredInt adds ~0.1 ms per request — negligible in most cases. Disable only if you have no credential-type policies configured, to avoid the overhead of loading the bloom filter into memory at startup.

Confidence threshold tuning

Each tier produces a confidence score for each detected entity. Entities below the configured threshold are not promoted to the next tier:

# Tier 1 → Tier 2 promotion threshold (default: 0.5)
TIER1_PROMOTION_THRESHOLD=0.7

# Tier 2 → Tier 3 promotion threshold (default: 0.6)
TIER2_PROMOTION_THRESHOLD=0.75

Raising the promotion threshold means fewer entities proceed to the next (more expensive) tier, which reduces latency. The trade-off is reduced recall — some true positives that would have been confirmed by DeBERTa are dropped earlier.

Caution

Raising thresholds to improve performance without re-validating accuracy is a common mistake. Any threshold change must be evaluated against your labeled test corpus using the accuracy harness. See DLP Accuracy for the validation workflow.

Pattern selection

If your use case only involves a subset of entity types, disable patterns for entity types you do not need:

# Disable entity types not relevant to your policy
DLP_ENTITY_TYPES=credit_card,ssn,api_key

Reducing the active entity type set shrinks the regex pattern bank and reduces the number of NER model heads that run, improving both latency and memory usage. See DLP Detection Hierarchy for the full list of supported entity types and their tier assignments.

---

5. Caching strategies

Outpost deliberately avoids result caching for scanned content — caching DLP decisions against content hashes would create a privacy risk (the cache itself becomes a sensitive data store). All caching is model-level and happens at startup.

What is cached at startup

Component	When loaded	Memory footprint
DeBERTa model weights	Process start	~4 GB (FP16/BF16) or ~8 GB (FP32)
NER model weights	Process start	~500 MB
Compiled regex pattern bank	Process start	~50 MB
CredInt bloom filter	Process start	~200 MB

All of these are loaded once and held in memory for the lifetime of the process. There is no lazy loading — if startup is slow, it is loading these assets.

Memory footprint by configuration

Configuration	Approximate memory
DeBERTa disabled, CredInt disabled	~600 MB
DeBERTa disabled, CredInt enabled	~800 MB
DeBERTa FP16/BF16, CredInt enabled	~4.5 GB
DeBERTa FP32, CredInt enabled	~8.5 GB

Note

These figures are for model memory only. Add OS overhead, the Go runtime, and connection buffers — allocate at least 1 GB headroom above the model footprint in your container memory limit.

Precision configuration

# FP16 (recommended for GPU — halves VRAM, minimal accuracy loss)
DEBERTA_PRECISION=fp16

# BF16 (alternative for Ampere+ GPUs — better numerical stability than FP16)
DEBERTA_PRECISION=bf16

# FP32 (default — full precision, double the memory)
DEBERTA_PRECISION=fp32

FP16 or BF16 is recommended for all production GPU deployments. The accuracy difference versus FP32 is negligible for entity detection tasks.

---

6. Benchmarking

Running the accuracy harness in benchmark mode

outpost-accuracy-harness \
  --target http://localhost:8080 \
  --corpus ./test-corpus/ \
  --benchmark-mode \
  --concurrency 10 \
  --duration 60s \
  --output benchmark-$(date +%Y%m%d-%H%M).json

Key flags:

Flag	Description
--benchmark-mode	Disables accuracy scoring; focuses on throughput and latency metrics
--concurrency N	Number of parallel request senders (simulate N concurrent clients)
--duration Ns	How long to run the load before reporting
--ramp-up 10s	Ramp concurrency up over N seconds (avoids cold-start skew)
--percentiles 50,95,99	Which latency percentiles to report

Example benchmark output

Arbitex Outpost Benchmark Report
=================================
Target:          http://localhost:8080
Duration:        60s
Concurrency:     10
Total requests:  5,842
Throughput:      97.4 req/s

Latency percentiles (end-to-end):
  p50:   14.2 ms
  p95:   22.8 ms
  p99:   38.1 ms
  p99.9: 91.4 ms
  max:   142.3 ms

Per-tier breakdown (p95):
  Tier 1 (regex):    1.1 ms
  Tier 2 (NER):      13.4 ms
  Tier 3 (DeBERTa):  7.8 ms
  CredInt:           0.1 ms

Error rate:      0.00%
Queue saturation: 0 requests dropped (0 HTTP 429)

Interpreting latency percentiles

Metric	Healthy	Degraded	Action
p50	< 20 ms (GPU) / < 80 ms (CPU)	> 40 ms (GPU) / > 150 ms (CPU)	Check tier breakdown; likely DeBERTa
p95	< 35 ms (GPU) / < 120 ms (CPU)	> 3× p50	Queue contention or GC pressure
p99	< 60 ms (GPU) / < 200 ms (CPU)	> 5× p50	Worker starvation or burst queuing
p99 / p50 ratio	< 3×	> 5×	High tail latency; check backpressure

A healthy deployment shows a tight p50–p99 spread. A wide spread (p99 more than 5× p50) indicates queueing or garbage collection pauses rather than a model inference problem.

Tip

Always benchmark at realistic concurrency levels. A single-client benchmark will show excellent p99 but will not reveal queueing effects that appear under production load.

---

7. Bottleneck identification

Prometheus metrics

Key metrics for performance diagnosis:

Metric	Type	What it tells you
outpost_scan_duration_seconds	Histogram	End-to-end scan latency; use {quantile="0.99"} for tail latency
outpost_scan_tier_duration_seconds	Histogram	Per-tier latency breakdown (labels: `tier=1
outpost_active_connections	Gauge	Current in-flight requests
outpost_request_queue_depth	Gauge	Requests waiting for a worker
outpost_http_requests_total	Counter	Total requests by status code
outpost_gpu_memory_used_bytes	Gauge	GPU VRAM consumption (GPU mode only)
outpost_scan_token_count	Histogram	Token distribution of scanned payloads

Diagnosis patterns:

• High outpost_scan_duration_seconds p99, high outpost_scan_tier_duration_seconds{tier="3"} p99: DeBERTa is the bottleneck. Options: switch to GPU, enable FP16, raise TIER2_PROMOTION_THRESHOLD, or disable DeBERTa if accuracy permits.

• High outpost_active_connections near MAX_CONCURRENT_REQUESTS: connection limit is the bottleneck. Increase MAX_CONCURRENT_REQUESTS and add workers, or scale horizontally.

• High queue depth, low CPU utilization: workers are blocked (likely on GPU synchronization or I/O). Reduce WORKER_THREADS to match available GPU streams, or investigate lock contention.

• Increasing outpost_gpu_memory_used_bytes over time: GPU memory leak. Restart pod and file a support issue. Use FP16 to reduce steady-state VRAM and give more headroom before OOM.

Structured log signals

Outpost emits structured JSON logs. Filter on event=scan_completed for per-request performance data:

{
  "event": "scan_completed",
  "request_id": "req_01HXYZ",
  "duration_ms": 18.4,
  "tier1_ms": 0.9,
  "tier2_ms": 10.2,
  "tier3_ms": 7.1,
  "credint_ms": 0.2,
  "entities_found": 2,
  "tokens": 387,
  "worker_id": 3
}

Aggregate duration_ms and per-tier fields with your log analytics platform to identify slow workers, payload size outliers, or tier-specific regressions.

---

8. Docker and Kubernetes tuning

Container resource configuration

Docker

# docker-compose.yml — GPU production
services:
  outpost:
    image: arbitex/outpost:latest
    environment:
      INFERENCE_DEVICE: cuda
      WORKER_THREADS: "6"
      MAX_CONCURRENT_REQUESTS: "100"
      DEBERTA_PRECISION: fp16
    deploy:
      resources:
        limits:
          memory: 12G
          cpus: "8"
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 60s

# docker-compose.yml — CPU-only dev/trial
services:
  outpost:
    image: arbitex/outpost:latest
    environment:
      INFERENCE_DEVICE: cpu
      DEBERTA_ENABLED: "false"
      WORKER_THREADS: "4"
      MAX_CONCURRENT_REQUESTS: "50"
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: "4"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 30s

Kubernetes

# outpost-deployment.yaml — GPU production
apiVersion: apps/v1
kind: Deployment
metadata:
  name: outpost
  namespace: arbitex
spec:
  replicas: 2
  selector:
    matchLabels:
      app: outpost
  template:
    metadata:
      labels:
        app: outpost
    spec:
      containers:
        - name: outpost
          image: arbitex/outpost:latest
          env:
            - name: INFERENCE_DEVICE
              value: cuda
            - name: WORKER_THREADS
              value: "6"
            - name: MAX_CONCURRENT_REQUESTS
              value: "100"
            - name: DEBERTA_PRECISION
              value: fp16
          resources:
            requests:
              memory: "10Gi"
              cpu: "4"
              nvidia.com/gpu: "1"
            limits:
              memory: "12Gi"
              cpu: "8"
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 30
            failureThreshold: 3
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

# outpost-deployment.yaml — CPU-only
apiVersion: apps/v1
kind: Deployment
metadata:
  name: outpost-cpu
  namespace: arbitex
spec:
  replicas: 4
  selector:
    matchLabels:
      app: outpost-cpu
  template:
    metadata:
      labels:
        app: outpost-cpu
    spec:
      containers:
        - name: outpost
          image: arbitex/outpost:latest
          env:
            - name: INFERENCE_DEVICE
              value: cpu
            - name: DEBERTA_ENABLED
              value: "false"
            - name: WORKER_THREADS
              value: "7"
            - name: MAX_CONCURRENT_REQUESTS
              value: "100"
          resources:
            requests:
              memory: "1Gi"
              cpu: "4"
            limits:
              memory: "2Gi"
              cpu: "8"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 45
            periodSeconds: 30

CPU pinning

For NER and DeBERTa workers on CPU-only deployments, CPU pinning reduces cache thrashing and improves latency consistency:

# Kubernetes: use CPU Manager static policy
# Node must be configured with --cpu-manager-policy=static
resources:
  requests:
    cpu: "8"    # Integer CPU request required for static policy
  limits:
    cpu: "8"

Alternatively, use cpuset in Docker:

docker run --cpuset-cpus="0-7" arbitex/outpost:latest

GPU device assignment

Outpost uses the NVIDIA device plugin for Kubernetes. Ensure the plugin is installed on GPU nodes:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml

For multi-GPU hosts where you want to assign a specific GPU to an Outpost instance:

# Docker: assign specific GPU by index or UUID
docker run --gpus '"device=0"' arbitex/outpost:latest

# Kubernetes: GPU assignment is handled by the device plugin scheduler
# Do not set CUDA_VISIBLE_DEVICES manually in K8s — let the plugin manage it

Health check tuning

The /health endpoint returns 200 only after all models are loaded and the worker pool is ready. Configure probes accordingly:

• Readiness probe: use /health, short interval (10s), moderate initialDelaySeconds (60s for GPU, 30s for CPU-only). This ensures the pod is not added to the load balancer until Outpost is ready to serve.
• Liveness probe: use /health, longer interval (30s), higher initialDelaySeconds (90s for GPU). Liveness should only restart genuinely hung processes — not slow-starting ones.

Caution

Setting livenessProbe.initialDelaySeconds too low on GPU deployments causes restart loops during model loading. DeBERTa in FP16 can take 30–45 seconds to load on first startup.

Horizontal Pod Autoscaler

Note

HPA is appropriate for CPU-bound (regex/NER-heavy) deployments. For GPU-bound DeBERTa deployments, HPA on CPU utilization is unreliable — scale based on outpost_request_queue_depth via KEDA or a custom metrics adapter instead.

# HPA for CPU-only (DeBERTa disabled) deployment
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: outpost-cpu-hpa
  namespace: arbitex
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: outpost-cpu
  minReplicas: 2
  maxReplicas: 16
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

# KEDA ScaledObject for GPU deployment — scale on queue depth
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: outpost-gpu-scaler
  namespace: arbitex
spec:
  scaleTargetRef:
    name: outpost
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: outpost_request_queue_depth
        threshold: "10"
        query: sum(outpost_request_queue_depth{namespace="arbitex"})

---

9. Common performance issues

Symptom	Likely cause	Fix
High p99 latency (> 5× p50)	Request queuing behind saturated workers	Increase WORKER_THREADS; increase MAX_CONCURRENT_REQUESTS; scale horizontally
High p99 latency, low queue depth	DeBERTa CPU-bound	Switch to GPU; enable ONNX INT8; raise TIER2_PROMOTION_THRESHOLD
Throughput plateau despite adding workers	GPU memory contention from too many concurrent GPU threads	Reduce WORKER_THREADS to match GPU stream count (typically 4–8)
Throughput plateau on CPU-only	Single-core saturation on NER	Increase WORKER_THREADS; pin workers to dedicated cores; scale horizontally
Memory growth over time	Model weight duplication (multiple processes sharing same node)	Use single Outpost process per GPU; verify no worker process leaks
GPU OOM at startup	FP32 model + insufficient VRAM	Switch to DEBERTA_PRECISION=fp16; upgrade to GPU with ≥ 16 GB VRAM
GPU OOM under load	Too many concurrent GPU inference calls	Reduce WORKER_THREADS; add MAX_CONCURRENT_REQUESTS cap
Slow startup (> 2 min)	Large model loading from slow storage	Use fast local NVMe for model cache; set MODEL_CACHE_DIR to a local path, not NFS
Erratic p95 (spiky)	GC pauses from large request payloads	Reduce max payload size via MAX_REQUEST_BYTES; monitor go_gc_duration_seconds
429 errors under moderate load	MAX_CONCURRENT_REQUESTS set too low	Raise limit in proportion to WORKER_THREADS; monitor queue depth
NER tier slow relative to DeBERTa	Regex pattern bank too large	Restrict active entity types with DLP_ENTITY_TYPES; audit pattern count

Tip

Most production performance issues are caused by misconfigured resource limits — either too low (causing OOM kills or CPU throttling) or too high (overprovisioning). Profile with the accuracy harness under realistic load before committing to a resource configuration.

---

For further reference, see Outpost Operations for general operational procedures, DLP Accuracy for accuracy validation workflows, DLP Detection Hierarchy for tier architecture details, and Deployment Topologies for multi-instance scaling patterns.