Outpost operations guide

This guide covers the full operational lifecycle of Arbitex Outpost: deployment, configuration, health monitoring, DLP pipeline management, GeoIP setup, certificate management, software updates, CLI usage, and troubleshooting. For architectural context, see Outpost Architecture. For initial deployment steps, see Outpost Deployment.

---

Deployment options

Docker (guided installer)

The install.sh installer is the recommended starting point for new deployments. It walks through six steps:

1. Prerequisite check — Docker 24+, Compose V2, curl, 5 GB free disk, port 8300 available
2. Configuration — prompts for OUTPOST_ID, PLATFORM_MANAGEMENT_URL, OUTPOST_API_KEY, and OUTPOST_EMERGENCY_ADMIN_KEY
3. Certificate download — retrieves mTLS bundle from the platform
4. Write .env — produces a validated environment file from .env.example
5. Validate — runs outpost validate-config against the new configuration
6. Start — launches the stack via docker compose -f docker-compose.outpost.yml up -d and polls /health for 60 seconds

curl -fsSL https://install.arbitex.ai/outpost | bash

For air-gap environments, set OUTPOST_AIRGAP=true before running the installer and supply a pre-fetched bundle:

OUTPOST_AIRGAP=true bash install.sh

Docker Compose (production)

The production compose file docker-compose.outpost.yml applies hardened defaults:

services:
  outpost:
    image: arbitex/outpost:latest
    ports:
      - "0.0.0.0:8300:8300"     # AI proxy — exposed to clients
      - "127.0.0.1:8301:8301"   # Admin API — loopback only
    read_only: true
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          memory: 2G
        reservations:
          memory: 512M
    tmpfs:
      - /tmp
    volumes:
      - ./certs:/app/certs:ro
      - outpost_policy:/app/policy_cache
      - outpost_audit:/app/audit_buffer
    env_file: .env
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8300/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

volumes:
  outpost_policy:
  outpost_audit:

Key security properties:

• Port 8301 (admin API) binds to 127.0.0.1 only — never expose it externally.
• read_only: true and no-new-privileges prevent filesystem and privilege escalation.
• Named volumes persist policy cache and audit buffer across container restarts.
• The container runs as user nonroot (uid/gid 65532) with tini as PID 1 for signal forwarding.

System requirements

Resource	Minimum	Recommended
Python	3.12+	3.12+
Docker Engine	24+ with Compose V2	Latest stable
Memory	512 MB reserved	2 GB limit
Disk	5 GB free	20 GB+ for audit buffer + models
Ports	8300 (proxy), 8301 (admin)	Same

Kubernetes / Helm

The Helm chart is at charts/arbitex-outpost/. Key values:

Value	Description
outpost.id	Unique outpost identifier (required)
outpost.platformManagementUrl	Management plane URL (required)
outpost.auditHmacKey	HMAC key for audit signing (required)
certs.existingSecret	Kubernetes secret containing mTLS cert bundle
autoscaling.enabled	Enable HPA for the outpost deployment
podDisruptionBudget.minAvailable	Minimum available pods during disruptions

helm upgrade --install arbitex-outpost charts/arbitex-outpost/ \
  --set outpost.id=op-prod-us-east-1 \
  --set outpost.platformManagementUrl=https://mgmt.arbitex.ai \
  --set outpost.auditHmacKey=<secret> \
  --set certs.existingSecret=arbitex-outpost-tls

---

Configuration reference

Outpost uses 12-factor environment variable injection. All settings are loaded from environment variables, with optional config file support via OUTPOST_CONFIG_FILE (default: /etc/outpost/config.env). Supported file formats: .env/.conf (KEY=VALUE) and .yaml/.yml (flat or nested, auto-detected).

Required variables

Variable	Description
OUTPOST_ID	Unique identifier for this outpost instance (UUID from Cloud portal)
PLATFORM_MANAGEMENT_URL	Base URL of the Arbitex management plane (skipped in air-gap)
AUDIT_HMAC_KEY	HMAC-SHA256 key for audit log chain integrity. Required.

Startup fails immediately if AUDIT_HMAC_KEY is empty, or if POLICY_HMAC_KEY is empty and INSECURE_SKIP_HMAC is not set.

Identity and registration

Variable	Default	Description
ORG_ID	""	Organisation UUID. Required for heartbeat/API paths.
OUTPOST_API_KEY	""	Shared secret for authenticating callers to /v1/chat/completions

mTLS certificates

Variable	Default	Description
OUTPOST_CERT_PATH	certs/outpost.pem	Path to the outpost mTLS certificate
OUTPOST_KEY_PATH	certs/outpost.key	Path to the outpost mTLS private key
OUTPOST_CA_PATH	certs/ca.pem	Path to the platform CA certificate
CERT_BUNDLE_AUTO_DOWNLOAD	false	Download cert bundle from platform at startup
CERT_BUNDLE_SIGNING_KEY	""	Hex HMAC-SHA256 key for cert bundle signature verification
CERT_AUTO_ROTATE	false	Enable automatic cert rotation via admin API
CERT_ROTATION_THRESHOLD_DAYS	30	Days before expiry to flag needs_rotation=true

TLS server configuration

Variable	Default	Description
TLS_MIN_VERSION	TLSv1.2	Minimum TLS version
TLS_MAX_VERSION	TLSv1.3	Maximum TLS version
TLS_VERIFY_CLIENT	false	Require client TLS certificates (mTLS)
TLS_CA_CERT_PATH	—	Path to CA cert for verifying client certs
TLS_SERVER_CERT_PATH	—	Path to server TLS certificate
TLS_SERVER_KEY_PATH	—	Path to server TLS private key

Policy

Variable	Default	Description
POLICY_HMAC_KEY	""	HMAC-SHA256 key for verifying signed policy bundles. Required in production.
INSECURE_SKIP_HMAC	false	DEV ONLY: skip POLICY_HMAC_KEY requirement
POLICY_BUNDLE_VERIFY	true	Enable policy bundle verification
POLICY_SYNC_INTERVAL	60	Policy sync interval in seconds
POLICY_CACHE_ENABLED	true	Enable TTL-based in-memory policy bundle cache
POLICY_CACHE_TTL_SECONDS	30	Policy cache TTL in seconds
BUDGET_ENFORCEMENT_ENABLED	true	Enforce dollar budget caps from policy

Admin API

Variable	Default	Description
OUTPOST_EMERGENCY_ADMIN_KEY	""	Bearer token for the admin API on port 8301
ADMIN_PORT	8301	Admin API listen port
ADMIN_JWT_SECRET	""	HMAC-SHA256 secret for signing admin JWTs
ADMIN_JWT_EXPIRY_SECONDS	3600	Admin JWT token expiry
REPLAY_PROTECTION_ENABLED	false	Enable request replay protection on state-changing admin endpoints
REPLAY_PROTECTION_TTL_SECONDS	60	TTL for replay protection request ID cache

Authentication

Variable	Default	Description
OAUTH_JWT_PUBLIC_KEY	""	PEM-encoded RSA public key for OAuth JWT validation
OAUTH_JWKS_URL	""	JWKS endpoint for fetching OAuth JWT public keys
OAUTH_JWKS_CACHE_TTL	300	JWKS key cache TTL in seconds
OAUTH_SCOPE_ENFORCEMENT	false	Enable OAuth scope enforcement

Rate limiting

Variable	Default	Description
RATE_LIMIT_ENABLED	true	Enable per-route rate limiting
RATE_LIMIT_REQUESTS_PER_MINUTE	60	Sustained rate limit for /v1/chat/completions (token bucket)
RATE_LIMIT_BURST	10	Token bucket burst capacity
RATE_LIMIT_SCAN_PER_MINUTE	120	Rate limit for /v1/scan
RATE_LIMIT_ADMIN_PER_MINUTE	30	Rate limit for /admin/*
RATE_LIMIT_DEFAULT_PER_MINUTE	60	Default rate limit for all other endpoints

DLP configuration

Variable	Default	Description
DLP_ENABLED	true	Enable DLP scanning pipeline (Tier 1 regex)
DLP_NER_ENABLED	true	Enable Tier 2 NER scanning (spaCy)
DLP_NER_MODEL	en_core_web_sm	spaCy NER model
DLP_NER_DEVICE	auto	Inference device: auto, cpu, cuda
DLP_DEBERTA_ENABLED	false	Enable Tier 3 DeBERTa ONNX inference
DEBERTA_MODEL_PATH	""	Path to DeBERTa ONNX model file
CREDINT_ENABLED	true	Enable Tier 4 CredInt bloom filter scanning
CREDINT_BLOOM_PATH	""	Path to CredInt bloom filter file (.bf)
CREDINT_CDN_URL	""	HTTPS URL for automatic bloom filter refresh
CREDINT_REFRESH_INTERVAL_SECONDS	86400	CDN refresh interval
CREDINT_KANON_ENABLED	false	Enable k-anonymity partial-hash check (requires network)
CREDINT_KANON_URL	https://api.pwnedpasswords.com/range	k-anonymity API URL
CUSTOM_REDACTION_PATTERNS	""	JSON array of {name, pattern, replacement, enabled} custom DLP patterns

Audit

Variable	Default	Description
AUDIT_HMAC_KEY	""	HMAC-SHA256 key for audit log chain integrity. Required.
AUDIT_CHAIN_ENABLED	false	Enable HMAC chaining on audit events
AUDIT_SYNC_INTERVAL_SECONDS	30	Interval between audit sync attempts to Platform
MAX_AUDIT_BUFFER_ENTRIES	100000	Maximum entries in audit buffer (ring buffer rotation)

Local storage

Variable	Default	Description
POLICY_CACHE_PATH	policy_cache/	Directory for cached policy bundles
AUDIT_BUFFER_PATH	audit_buffer/	Directory for local audit log buffer
BACKUP_DIR	""	Config backup snapshot directory (empty = in-memory ring buffer, last 10)
USAGE_DB_PATH	usage_data/usage.db	SQLite database for usage tracking
AUDIT_QUEUE_DB_PATH	audit_queue/audit_queue.db	SQLite database for local audit queue during degradation

Logging

Variable	Default	Description
LOG_LEVEL	info	Log verbosity: debug, info, warning, error, critical
LOG_FORMAT	text	Log output format: text or json
DEBUG	false	Enable debug mode

Lifecycle

Variable	Default	Description
SHUTDOWN_TIMEOUT	30	Seconds to wait for in-flight requests during graceful shutdown
DRAIN_TIMEOUT_SECONDS	30	Seconds to wait for active DLP scans on shutdown

Heartbeat

Variable	Default	Description
CLOUD_HEARTBEAT_URL	""	Cloud portal heartbeat endpoint base URL
CLOUD_HEARTBEAT_INTERVAL	60	Heartbeat interval in seconds

Circuit breakers

Variable	Default	Description
CIRCUIT_BREAKER_FAILURE_THRESHOLD	5	Consecutive failures before breaker opens
CIRCUIT_BREAKER_RECOVERY_TIMEOUT	30	Seconds breaker stays OPEN before HALF_OPEN
CIRCUIT_BREAKER_HALF_OPEN_MAX_REQUESTS	3	Max probe calls in HALF_OPEN state

Named breakers: policy_sync, heartbeat, audit_sync.

Connection pool

Variable	Default	Description
PLATFORM_MAX_CONNECTIONS	100	Max total connections in HTTP pool
PLATFORM_MAX_KEEPALIVE	20	Max keep-alive connections
PLATFORM_KEEPALIVE_EXPIRY	30	Keep-alive expiry in seconds

GeoIP

Variable	Default	Description
GEOIP_MMDB_PATH	""	Explicit path to MaxMind City MMDB
GEOIP_MMDB_FALLBACK_PATH	""	Legacy fallback MMDB path
GEOIP_MMDB_BUNDLED_PATH	/opt/outpost/geoip/GeoLite2-City.mmdb	Container-baked MMDB
GEOIP_DOWNLOAD_ON_START	false	Download MMDB at startup if not present
GEOIP_DOWNLOAD_URL	""	URL to download MMDB from
GEOIP_ANON_DB_PATH	""	Path to MaxMind Anonymous IP MMDB
GEOIP_ANON_BUNDLED_PATH	/opt/outpost/geoip/GeoIP2-Anonymous-IP.mmdb	Bundled Anonymous IP MMDB

Software updates

Variable	Default	Description
SOFTWARE_UPDATE_RELEASE_URL	""	URL for release manifest JSON
SOFTWARE_UPDATE_ED25519_KEY	""	Base64 Ed25519 public key (primary verification)
SOFTWARE_UPDATE_SIGNING_KEY	""	Alias for SOFTWARE_UPDATE_ED25519_KEY
SOFTWARE_UPDATE_STAGE_DIR	/tmp/outpost-update-stage	Staging directory for downloaded bundles

Air-gap

Variable	Default	Description
OUTPOST_AIRGAP	false	Enable air-gap mode
AIRGAP_POLICY_PATH	/opt/arbitex/policies	Directory with policy_bundle.json
AIRGAP_MODEL_PATH	/opt/arbitex/models	Directory with DeBERTa ONNX model

Log export and rotation

Variable	Default	Description
LOG_EXPORT_ENABLED	false	Enable structured audit log file export
LOG_EXPORT_PATH	/var/log/outpost/audit/	Export directory
LOG_EXPORT_FORMAT	jsonl	Format: jsonl or csv
LOG_EXPORT_ROTATION_MB	100	Max file size (MB) before rotation
LOG_EXPORT_MAX_FILES	10	Max rotated files to keep
LOG_ROTATION_ENABLED	false	Enable RotatingFileHandler for log files
LOG_MAX_BYTES	104857600	Max log file size in bytes before rotation (100 MB)
LOG_BACKUP_COUNT	5	Number of rotated log backup files

Disk monitoring

Variable	Default	Description
DISK_MONITOR_ENABLED	false	Enable background disk usage monitoring
DISK_MONITOR_PATH	/	Filesystem path to monitor
DISK_MONITOR_INTERVAL_SECONDS	300	Interval between disk checks (5 min)
DISK_WARN_THRESHOLD_PCT	75.0	Disk % that triggers WARNING
DISK_CRITICAL_THRESHOLD_PCT	90.0	Disk % that triggers CRITICAL

IP allowlist

Variable	Default	Description
IP_ALLOWLIST_ENABLED	false	Enable IP allowlisting
IP_ALLOWLIST_CIDRS	""	Comma-separated CIDR ranges
IP_ALLOWLIST_ADMIN_EXEMPT	true	/admin/* bypasses IP allowlist

SIEM direct sink

Variable	Default	Description
SIEM_DIRECT_ENABLED	false	Enable SIEM direct sink
SIEM_DIRECT_TYPE	splunk_hec	SIEM type: splunk_hec or syslog
SIEM_DIRECT_URL	""	SIEM endpoint URL
SIEM_DIRECT_TOKEN	""	Auth token for SIEM (Splunk HEC token)
SIEM_DIRECT_BUFFER_CAPACITY	10000	Ring buffer capacity (events)
SIEM_DIRECT_DEAD_LETTER_PATH	""	Dead-letter JSONL file path (empty=disabled)
SIEM_OUTPUT	syslog	Output format: syslog (RFC 5424) or cef

Monitoring exports

Variable	Default	Description
PUSHGATEWAY_ENABLED	false	Enable Prometheus Pushgateway push
PUSHGATEWAY_URL	""	Pushgateway base URL
PUSHGATEWAY_JOB	outpost	Pushgateway job label
PUSHGATEWAY_INTERVAL_SECONDS	60	Push interval
PUSHGATEWAY_INSTANCE	""	Instance label (default: hostname)
OTEL_ENABLED	false	Enable OTel distributed tracing
OTEL_EXPORTER_ENDPOINT	""	OTLP gRPC exporter endpoint
OTEL_SERVICE_NAME	arbitex-outpost	OTel service name

Body hash logging

Variable	Default	Description
BODY_HASH_LOGGING_ENABLED	false	Enable HMAC hashing of request/response bodies
BODY_HASH_ALGORITHM	sha256	Hash algorithm: sha256 or sha512
BODY_HASH_LOG_RESPONSE	false	Also hash response bodies

Request validation

Variable	Default	Description
MAX_REQUEST_BODY_MB	10	Max request body size in MB for /v1/chat/completions
REQUEST_MAX_BODY_BYTES	1048576	Max request body size in bytes for validation middleware (1 MB)

Multi-org and cluster

Variable	Default	Description
MULTI_ORG_MODE	false	Enable multi-org mode
MAX_ORGS_PER_OUTPOST	10	Max distinct orgs in multi-org mode
CLUSTER_PEERS	""	Comma-separated peer outpost admin URLs
CLUSTER_HEALTH_TIMEOUT	3.0	Timeout (seconds) for peer health queries
CLUSTER_HEALTH_AUTH_KEY	""	API key for authenticating to peer outposts

Plugins and webhooks

Variable	Default	Description
PLUGINS_ENABLED	false	Enable plugin system
PLUGIN_DIR	./plugins	Directory to scan for plugin .py files
WEBHOOK_EMIT_ENABLED	false	Enable webhook event emission on scan completion
WEBHOOK_EMIT_URL	""	Target URL for webhook delivery
WEBHOOK_EMIT_SECRET	""	HMAC-SHA256 secret for signing webhook payloads

Provider key encryption

Variable	Default	Description
PROVIDER_KEY_ENCRYPTION_KEY	""	Fernet key (base64-urlsafe, 32 bytes) for decrypting provider API keys from policy bundle

PROMPT hold

Variable	Default	Description
PROMPT_HOLD_TIMEOUT_SECONDS	300	Seconds to wait for admin approval before blocking
PROMPT_HOLD_QUEUE_MAX	100	Max pending holds (0=unlimited)
PROMPT_HOLD_OVERFLOW	reject	Queue overflow: reject or evict (drop oldest)
PROMPT_HOLD_TTL_SECONDS	86400	TTL for pending holds (24 h); auto-rejected after

Example .env

OUTPOST_ID=op-prod-us-east-1
ORG_ID=org-123
PLATFORM_MANAGEMENT_URL=https://mgmt.arbitex.ai
AUDIT_HMAC_KEY=<base64-encoded-key>
POLICY_HMAC_KEY=<base64-encoded-key>
OUTPOST_API_KEY=<shared-secret>
OUTPOST_EMERGENCY_ADMIN_KEY=<admin-bearer-token>
OUTPOST_CERT_PATH=/app/certs/outpost.pem
OUTPOST_KEY_PATH=/app/certs/outpost.key
OUTPOST_CA_PATH=/app/certs/ca.pem
LOG_LEVEL=info
LOG_FORMAT=json
SHUTDOWN_TIMEOUT=30
POLICY_SYNC_INTERVAL=60

---

Startup validation

outpost validate-config (or outpost config validate) runs all configuration checks. Use --strict to treat warnings as failures (exit code 1 on any warn or fail).

Check	What it verifies	Fail criteria
env_vars	OUTPOST_ID, PLATFORM_MANAGEMENT_URL (skipped in air-gap), AUDIT_HMAC_KEY non-empty	Any required var missing or empty
mtls_certs	OUTPOST_CERT_PATH, OUTPOST_KEY_PATH, OUTPOST_CA_PATH exist, are PEM, not expired	File missing, unreadable, or cert expired. Warns at < 30 days.
platform_reach	Platform management URL responds to HTTP GET	Non-2xx or network error. Skipped in air-gap.
geoip_mmdb	GeoIP MMDB file readable	File configured but missing/invalid. Skipped when no path configured.
policy_bundle	Policy bundle signature valid	Signature mismatch or key missing. Skipped when verification disabled.
port_availability	Ports 8300 and 8301 not already bound	Either port in use
deberta_model	DeBERTa ONNX model file valid	DLP_DEBERTA_ENABLED=true but path missing or invalid
software_update	SOFTWARE_UPDATE_RELEASE_URL + Ed25519 key config valid	Key set without URL or URL set without key
dlp_pipeline	DLP tier dependencies consistent with enabled flags	Dependency conflict (e.g., DeBERTa enabled without NER)
heartbeat	Heartbeat URL format and interval sanity	Invalid URL format. Skipped in air-gap.
config_file_permissions	Config file permissions check	World-readable sensitive files
tls_cert_readable	TLS server cert readable	Certificate configured but unreadable
dlp_regex_compilation	DLP regex patterns compile without errors	Invalid regex pattern

Exit codes: 0 = all pass, 1 = at least one fail, 2 = warnings only (no failures).

outpost validate-config --json
outpost config validate --strict

Security fail-fast at startup

In non-air-gap mode, startup calls validate_security_requirements() — this causes immediate SystemExit(1) if POLICY_HMAC_KEY is unset and INSECURE_SKIP_HMAC is not set.

---

Certificate management

Outpost uses two categories of cryptographic material: mTLS certificates for authenticating connections to the management plane, and Ed25519 signing keys for verifying policy bundles and software update packages. Both require active lifecycle management.

mTLS certificate lifecycle

mTLS certificates are issued per-outpost by the Arbitex platform CA. They authenticate every outpost-to-platform call including heartbeat, policy sync, and audit upload.

Initial download

When CERT_BUNDLE_AUTO_DOWNLOAD=true, the outpost fetches its certificate bundle from the platform at startup using the OUTPOST_API_KEY for bootstrap authentication. The bundle is written to the paths specified by OUTPOST_CERT_PATH, OUTPOST_KEY_PATH, and OUTPOST_CA_PATH.

# Manual cert bundle download using the admin API
curl -s \
  -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY" \
  http://127.0.0.1:8301/admin/certs/download \
  | jq .

Auto-renewal

When CERT_AUTO_ROTATE=true, the outpost monitors its own certificate expiry. When fewer than CERT_ROTATION_THRESHOLD_DAYS (default: 30) remain, the outpost:

1. Generates a new CSR internally
2. Submits the CSR to POST /v1/orgs/{org_id}/outposts/{outpost_id}/certs/renew on the management plane
3. Writes the returned certificate to OUTPOST_CERT_PATH
4. Reloads the TLS listener without dropping active connections

Auto-renewal requires the outpost to be online and able to reach the management plane. For air-gap deployments, see Air-gap cert management below.

Expiry monitoring via heartbeat

The enhanced heartbeat payload includes a cert_expiry field. The platform fires a warning when expiry is within 30 days and a critical alert when within 7 days. The /health endpoint surfaces this in the warnings array.

Certificate status CLI

# Check cert expiry and chain status
outpost certs status
outpost certs status --json

# Verify leaf → CA chain (structural check)
outpost certs verify
outpost certs verify --json

Warning thresholds: ≤ 30 days = INFO, ≤ 14 days = WARNING, ≤ 7 days = WARNING, ≤ 1 day = CRITICAL. Exit code 1 if any cert is critical or expired.

Ed25519 signing keys

Ed25519 keys serve two distinct purposes:

Key	Variable	Purpose
Policy bundle signing key	POLICY_HMAC_KEY	Verifies signatures on policy bundles fetched from the platform
Software update signing key	SOFTWARE_UPDATE_ED25519_KEY	Verifies signatures on software update packages

Both keys are public keys distributed by Arbitex and pinned in configuration.

Certificate rotation procedure (zero-downtime)

Use this procedure to rotate the mTLS certificate without dropping active proxy connections.

1. Issue the new certificate from the platform admin console or via API:

   curl -s -X POST \
     -H "Authorization: Bearer $PLATFORM_API_KEY" \
     "https://mgmt.arbitex.ai/v1/orgs/org-123/outposts/op-prod-us-east-1/certs/issue" \
     | jq .

2. Download the new cert bundle to a staging location on the outpost host:

   scp new-outpost.pem outpost-host:/app/certs/outpost-new.pem
   scp new-outpost.key outpost-host:/app/certs/outpost-new.key

3. Validate the new certificate before activating:

   openssl verify -CAfile /app/certs/ca.pem /app/certs/outpost-new.pem
   openssl x509 -noout -dates -in /app/certs/outpost-new.pem

4. Atomically swap the certificate files:

   mv /app/certs/outpost.pem /app/certs/outpost.pem.bak
   mv /app/certs/outpost.key /app/certs/outpost.key.bak
   mv /app/certs/outpost-new.pem /app/certs/outpost.pem
   mv /app/certs/outpost-new.key /app/certs/outpost.key

5. Send SIGHUP to reload the TLS config without restarting:

   docker kill --signal=SIGHUP $(docker compose ps -q outpost)

6. Verify the rotation succeeded:

   curl -s http://localhost:8300/health | jq '.components.certificates'
   outpost certs status

7. Remove backup files once the new cert is confirmed healthy:

   rm /app/certs/outpost.pem.bak /app/certs/outpost.key.bak

Caution

SIGHUP reloads TLS certificates and hot-reloadable configuration but does not restart listen ports. If the new certificate fails validation, the outpost continues serving with the previous certificate. Always verify /health after sending SIGHUP.

Air-gap cert management

In air-gap deployments, auto-renewal is disabled and all certificate operations are performed manually:

1. Issue the new certificate from the Arbitex platform console on an internet-connected workstation.
2. Transfer the certificate bundle through your approved transfer channel (USB, one-way transfer device, etc.).
3. Place the files at the paths matching OUTPOST_CERT_PATH, OUTPOST_KEY_PATH, and OUTPOST_CA_PATH.
4. Send SIGHUP or restart the container to activate the new certificate.

# Check cert expiry locally (air-gap)
outpost certs status
openssl x509 -noout -enddate -in /app/certs/outpost.pem

---

Health monitoring

Outpost exposes three HTTP endpoints on port 8300 for health and readiness signaling.

Endpoints

Endpoint	Purpose	Success	Failure
GET /health	Component-level status	200 OK	503 Service Unavailable
GET /ready	Readiness probe — blocks traffic until critical components are up	200 OK	503 Service Unavailable
GET /live	Liveness probe — always returns 200 if the event loop is alive	200 OK	N/A

/health response structure

{
  "status": "ok",
  "overall_status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 86400.0,
  "mode": "single",
  "components": {
    "proxy":        { "status": "ok" },
    "dlp":          { "status": "ok" },
    "policy_sync":  { "status": "ok" },
    "heartbeat":    { "status": "ok" },
    "config":       { "status": "ok" },
    "audit":        { "status": "ok" },
    "geoip":        { "status": "ok" },
    "certificates": { "status": "ok" }
  },
  "warnings": [],
  "dlp_pipeline": "ok",
  "circuit_breaker_open": false,
  "disk_ok": true,
  "cert_expiry_days": 180,
  "audit_chain_enabled": false,
  "pushgateway_active": false
}

status values: ok, degraded, unhealthy. overall_status mapping: ok → healthy, degraded → degraded, unhealthy → unhealthy. HTTP 503 when overall_status == "unhealthy".

cert_expiry_days is the minimum days remaining across all certs; -1 if no certs found.

/ready critical components

The readiness probe only passes when all four critical components initialize successfully:

• policy_bundle — policy loaded and signature verified
• dlp_pipeline — at least Tier 1 regex active
• audit_logger — audit writer initialized
• proxy_router — proxy routing table built

Optional components (degraded but not blocking): heartbeat, policy_sync. The probe also checks whether critical circuit breakers (policy_sync, audit_sync) are OPEN — if so, the status is degraded.

{
  "status": "ready",
  "outpost_id": "op-prod-us-east-1",
  "policy_version": "v42",
  "components": {
    "policy_bundle": "ok",
    "dlp_pipeline": "ok",
    "audit_logger": "ok",
    "proxy_router": "ok",
    "heartbeat": "ok",
    "policy_sync": "ok"
  },
  "circuit_breakers_open": []
}

/live response

{ "status": "alive", "version": "0.1.0" }

Component check details

Component	Healthy	Degraded	Unhealthy
proxy	Router initialized	—	Router not initialized
dlp	All enabled tiers active	—	No tiers active
policy_sync	Bundle age ≤ 300 s	Bundle age 301–3600 s	Bundle age > 3600 s
heartbeat	Sender initialized	—	Not initialized
config	All checks pass	Non-critical check failure	Critical check failure
audit	Writer initialized	—	Writer not initialized
geoip	All configured DBs loaded	One DB unavailable	No DBs loaded
certificates	All certs valid	Expiry within 30 days	Cert expired or critical

Admin health endpoints

The admin API on port 8301 provides additional health endpoints:

Endpoint	Description
GET /admin/api/health	Full health for admin consumers (same detailed schema)
GET /admin/api/health/cluster	Aggregated health from all cluster peers
GET /admin/api/health/components	Per-component health details
GET /admin/api/health/summary	Brief health summary
GET /admin/diagnostics	Diagnostic data via admin API

---

Heartbeat

The HeartbeatSender maintains the live connection between each outpost and the platform.

Behavior

• Default interval: 60 seconds (CLOUD_HEARTBEAT_INTERVAL)
• Backoff: exponential with jitter, capped at 900 seconds
• On graceful shutdown: sends a final heartbeat with shutting_down: true
• Disabled in air-gap mode (no cloud heartbeat URL)

Heartbeat payload

{
  "version": "0.1.0",
  "uptime": 86400,
  "policy_version": "v20260314-001",
  "last_sync_at": "2026-03-14T12:00:00Z",
  "dlp_model_version": "en_core_web_sm-3.7.1",
  "pending_audit_events": 0,
  "tier3_active": false,
  "resource_usage": {
    "cpu_percent": 12.4,
    "memory_percent": 38.1,
    "disk_percent": 21.7
  },
  "cert_expiry": "2026-09-01T00:00:00Z",
  "dlp_tiers_active": [1, 2, 4]
}

Version mismatch warning

The platform response includes a latest_version field. If the running version differs, the outpost logs a warning and surfaces it in /health under warnings. No automatic update is triggered.

---

DLP pipeline configuration

The DLP pipeline runs a 5-tier cascade. See DLP Pipeline Architecture for detailed design.

Tiers

Tier	Technology	Enable variable	Default
1	Regex pattern matching	DLP_ENABLED	true
2	NER / spaCy	DLP_NER_ENABLED	true
3	DeBERTa ONNX inference	DLP_DEBERTA_ENABLED	false
4	CredInt bloom filter	CREDINT_ENABLED	true

Constraint: Tier 3 (DeBERTa) requires Tier 2 (NER) to be enabled.

Cascade flow

1. Tier 1 regex scan
2. If any result is BLOCK — terminate cascade, return block action
3. Tier 2 NER scan — merge findings with Tier 1
4. Tier 3 DeBERTa inference — contextual validation of NER findings
5. Tier 4 CredInt bloom filter check
6. Apply policy rules
7. Deduplicate findings
8. Resolve final action (priority: block > cancel > redact > log_only)

---

GeoIP setup

GeoIP resolution uses a 4-tier fallback chain to locate the MMDB database file.

Database resolution order

1. Explicit path via GEOIP_MMDB_PATH
2. Download on start if GEOIP_DOWNLOAD_ON_START=true (requires network; skipped in air-gap)
3. Bundled path: /opt/outpost/geoip/GeoLite2-City.mmdb
4. Fallback path via GEOIP_MMDB_FALLBACK_PATH

Download security

SSRF protection is applied to all GeoIP downloads:

• HTTPS required — HTTP URLs are rejected
• Allowed hosts restricted to download.maxmind.com by default

Updating the GeoIP database

GeoIP does not auto-update. To refresh:

1. Download the new MMDB file to the host
2. Replace the file at GEOIP_MMDB_PATH (or mounted volume path)
3. Restart the container or send SIGHUP

For air-gap: mount your GeoLite2-City.mmdb at the bundled path or set GEOIP_MMDB_PATH explicitly.

---

Software updates

The SoftwareUpdateManager provides verified, staged software updates with fail-closed signature verification.

Tip

For a customer-facing overview of how the Outpost keeps itself patched — the auto (default), notify, and manual update modes, maintenance windows, and canary rollout with automatic rollback — see Outpost update modes. The staged flow described below is the operator-driven mechanism behind the notify and manual modes.

Update states

idle → checking → available → downloading → staged
                                          → verify_failed
                                          → error

Updates move to staged and wait for operator action. There is no auto-apply.

Signature verification

Verification priority:

1. Ed25519 (primary) — SOFTWARE_UPDATE_ED25519_KEY (base64-encoded 32-byte public key). Downloads bundle tarball + detached .sig file, verifies with Ed25519.
2. HMAC-SHA256 (fallback) — POLICY_HMAC_KEY + hmac_signature in manifest.
3. Fail-closed — if neither verification method is available, the bundle is rejected: "No verification method available — configure SOFTWARE_UPDATE_ED25519_KEY or POLICY_HMAC_KEY (fail-closed)"

Update flow

1. POST /admin/api/updates/check — fetch manifest from SOFTWARE_UPDATE_RELEASE_URL, compare versions
2. POST /admin/api/updates/download — download, verify, extract to SOFTWARE_UPDATE_STAGE_DIR
3. Restart container — staged update applied on next start

Manifest schema:

{
  "version": "1.5.0",
  "download_url": "https://releases.arbitex.ai/outpost/outpost-1.5.0.tar.gz",
  "signature_url": "https://releases.arbitex.ai/outpost/outpost-1.5.0.tar.gz.sig",
  "hmac_signature": "...",
  "release_notes": "..."
}

Applying an update

# Check for available updates
outpost update check --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY

# Apply staged update
outpost update apply --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY

# Restart to activate
docker compose restart outpost

Air-gap updates

In air-gap mode, operators place bundles and signatures manually in SOFTWARE_UPDATE_STAGE_DIR:

# On the operator workstation (internet-connected):
# Download bundle and signature from the Arbitex release CDN

# On the air-gap host:
cp outpost-1.5.0.tar.gz /tmp/outpost-update-stage/
cp outpost-1.5.0.tar.gz.sig /tmp/outpost-update-stage/

# Check and apply
outpost update check --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY
outpost update apply --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY
docker compose restart outpost

Status: GET /admin/api/updates/status

---

Circuit breakers

Outpost wraps its three external dependencies — policy sync, heartbeat, and audit sync — in circuit breakers that prevent cascading failures when the management plane is unreachable.

State machine

Each circuit breaker follows a three-state machine:

          failure threshold reached
CLOSED ──────────────────────────────► OPEN
  ▲                                      │
  │                               recovery timeout
  │                                   elapses
  │                                      │
  │        all half-open calls pass      ▼
  └──────────────────────────────── HALF_OPEN
                  │
                  │ any half-open call fails
                  ▼
                OPEN (reset recovery timer)

• CLOSED — normal operation; all calls pass through
• OPEN — breaker tripped; calls are rejected immediately without attempting the operation
• HALF_OPEN — recovery probe; a limited number of calls are allowed through to test recovery

Per-breaker configuration

Circuit breaker	Failure threshold	Recovery timeout	Half-open max calls
policy_sync	5 failures	30 s	3
heartbeat	5 failures	30 s	3
audit_sync	5 failures	30 s	3

Configurable via CIRCUIT_BREAKER_FAILURE_THRESHOLD, CIRCUIT_BREAKER_RECOVERY_TIMEOUT, and CIRCUIT_BREAKER_HALF_OPEN_MAX_REQUESTS.

Behavior when OPEN

Breaker	OPEN behavior	Impact
policy_sync	Policy sync suspended; outpost continues enforcing the last successfully loaded policy bundle	Stale policy risk; /health reports degraded when bundle age > 300 s, unhealthy > 3600 s
heartbeat	Heartbeat transmissions suspended; platform marks outpost as offline	No connectivity loss for proxied traffic; platform observability is lost
audit_sync	Audit upload suspended; events accumulate in the local audit buffer	No data loss — buffer absorbs events (up to MAX_AUDIT_BUFFER_ENTRIES = 100,000)

Caution

If policy_sync is OPEN for more than 1 hour (bundle age > 3600 s), the policy_sync component transitions to unhealthy and the overall /health status becomes unhealthy. Load balancers using the readiness probe will remove the outpost from rotation.

Metrics to monitor

Metric	What to watch for
outpost_circuit_breaker_state{breaker}	0=closed, 1=open, 2=half_open
outpost_circuit_breaker_trips_total{breaker}	Increasing count = repeated trips
outpost_heartbeat_consecutive_failures	> 5 means breaker is likely OPEN
outpost_policy_sync_age_seconds	Age increasing without bound = sync not recovering

curl -s http://localhost:8300/health | jq '{
  policy_sync: .components.policy_sync,
  heartbeat: .components.heartbeat,
  audit: .components.audit,
  circuit_breaker_open: .circuit_breaker_open
}'

---

Audit trail

Outpost maintains a tamper-evident local audit trail using HMAC chaining. The audit buffer is stored as HMAC-chained JSONL at {AUDIT_BUFFER_PATH}/audit.jsonl (default: audit_buffer/audit.jsonl), with a maximum of MAX_AUDIT_BUFFER_ENTRIES (default: 100,000) entries in a ring buffer.

HMAC chain

Every audit event is linked to the previous event by including the HMAC of the previous event’s data. This creates a chain where any modification to a historical event breaks all subsequent event signatures.

Event structure:

{
  "event_id": "evt-20260314-001823",
  "timestamp": "2026-03-14T14:23:01.482Z",
  "outpost_id": "op-prod-us-east-1",
  "org_id": "org-123",
  "event_type": "dlp_scan",
  "action": "block",
  "policy_id": "pol-cc-data",
  "prev_event_hash": "sha256:a3f9c8d2e1b4...",
  "payload": {
    "request_id": "req-abc123",
    "tier": 1,
    "finding_count": 2
  },
  "hmac": "sha256:7b2e1d9f4a..."
}

The hmac field is computed as:

HMAC-SHA256(AUDIT_HMAC_KEY, event_id + timestamp + prev_event_hash + canonical_payload)

Audit export

Use the CLI to export audit events with filtering and chain verification:

outpost audit export --hmac-key $AUDIT_HMAC_KEY

Flags:

Flag	Description
--hmac-key KEY	HMAC key (required; or set AUDIT_HMAC_KEY env var)
--audit-path DIR	Audit buffer directory (default: $AUDIT_BUFFER_PATH or audit_buffer)
--format, -f	Output format: json (default), csv, jsonl
--from, --since	Lower bound date filter (ISO 8601, inclusive)
--to, --until	Upper bound date filter (ISO 8601, inclusive)
--type TYPE	Event type filter; supports glob (e.g., dlp.*)
--output, -o	Output file path (default: stdout)
--verify	Run HMAC chain verification before export
--limit N	Limit to first N matching entries
--count	Print only the count of matching entries (no export)
--json	Output summary JSON object

Examples:

# Export all events as JSON
outpost audit export --hmac-key $AUDIT_HMAC_KEY

# Export a date range as CSV
outpost audit export --format csv --output /tmp/audit.csv \
  --from 2026-01-01 --to 2026-01-31 --hmac-key $AUDIT_HMAC_KEY

# Export DLP events only, with chain verification
outpost audit export --type 'dlp.*' --verify --json --hmac-key $AUDIT_HMAC_KEY

# Count auth events
outpost audit export --count --type 'auth.*' --hmac-key $AUDIT_HMAC_KEY

Chain verification

Verify the integrity of the entire audit chain:

outpost audit verify --hmac-key $AUDIT_HMAC_KEY
outpost audit verify --hmac-key $AUDIT_HMAC_KEY --json

JSON output:

{
  "valid": true,
  "total_entries": 12430,
  "first_break_at": null,
  "errors": []
}

Exit code 1 if chain is invalid.

Audit sync to platform

Audit events are uploaded to the platform in batches via:

POST /v1/orgs/{org_id}/outposts/{outpost_id}/audit/batch

• Batch size: up to 500 events per request
• Upload interval: every AUDIT_SYNC_INTERVAL_SECONDS (default: 30) when events are pending
• Retry: exponential backoff on transient HTTP errors
• Deduplication: the platform deduplicates by event_id; safe to retry
• Ordering: batches are uploaded in chain order

If batch upload fails after retries, the audit_sync circuit breaker records the failure. After 5 consecutive failures the breaker opens and upload is suspended. Events continue accumulating locally without loss.

Admin audit API

Endpoint	Description
GET /admin/audit/recent	Recent audit entries
GET /admin/audit/stats	Audit statistics
GET /admin/audit/verify	HMAC chain verification via API
GET /admin/audit/search	Search audit entries
GET /admin/api/audit-buffer	Audit buffer contents
GET /admin/audit-queue/status	Local SQLite queue status
POST /admin/audit-queue/flush	Flush queued events
DELETE /admin/audit-queue/purge	Purge queue

---

CLI reference

The outpost CLI is available as outpost (installed via pyproject.toml scripts).

Global flags

Flag	Description
--json	Output as machine-readable JSON
--admin-port PORT	Admin API port (default: 8301)
--admin-key KEY	Admin API bearer token (overrides OUTPOST_EMERGENCY_ADMIN_KEY)

Commands

outpost status

Display current outpost status including version, uptime, and component health.

outpost status
outpost status --json
outpost status --local  # gather from config/filesystem only, no admin API calls

outpost validate-config / outpost config validate

Run all configuration checks. See Startup validation for the full check table.

outpost validate-config
outpost validate-config --json
outpost config validate --strict  # treat warnings as failures

outpost config reload

Trigger configuration hot-reload via the admin API. Equivalent to sending SIGHUP.

outpost config reload --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY
outpost config reload --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY --dry-run  # diff only
outpost config reload --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY --json

outpost policy-show

Display the currently loaded policy bundle: version, hash, rule count, providers. Offline — reads from disk, no admin API calls.

outpost policy-show
outpost policy-show --json

outpost benchmark-run

Run a DLP pipeline benchmark to measure per-tier latency.

outpost benchmark-run
outpost benchmark-run --json --timeout 120

outpost update check / outpost update apply

Check for and apply software updates.

outpost update check --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY
outpost update apply --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY

outpost diagnostics

Collect a full diagnostic report: system info, config (redacted by default), DLP tier status, GeoIP, connectivity, config validation, TLS certs, metrics, and health. Operates entirely offline — no admin API calls.

outpost diagnostics
outpost diagnostics --json
outpost diagnostics --json --output-file diagnostics.json
outpost diagnostics --no-redact  # show API keys and secrets unmasked

Output sections:

Section	Contents
system	Python version, platform, hostname, PID, uptime
config	Outpost ID, air-gap flag, DLP enabled, proxy/admin ports, redacted secrets
dlp_tiers	Regex, NER, DeBERTa, CredInt — status and model info per tier
geoip	City DB and Anonymous IP DB — path, present, age
heartbeat	URL, interval, last heartbeat timestamp
connectivity	Platform reachability (TCP socket check, 5 s timeout; skipped in air-gap)
config_validation	Pass/fail/warn/skip counts, overall status, per-check details
tls	TLS cert chain summary
metrics	Runtime metrics snapshot
health	Component health summary

Human-readable output uses ANSI color on TTY (green=ok, yellow=warn, red=error).

outpost health export

Export current health status as JSON, suitable for external monitoring ingestion.

outpost health export
outpost health export --output health-report.json

Wraps diagnostics data with report_version and generated_at metadata.

outpost certs status

Show certificate subject, issuer, expiry, days remaining, and warning level. Exit 1 if any cert is critical or expired.

outpost certs status
outpost certs status --json

outpost certs verify

Verify leaf → CA chain structure (issuer/subject match, validity windows). Structural check — use openssl verify for full PKI validation.

outpost certs verify
outpost certs verify --json

outpost audit export

Export signed audit events with filtering. See Audit export for full flag reference.

outpost audit export --hmac-key $AUDIT_HMAC_KEY
outpost audit export --format csv --output audit.csv --hmac-key $AUDIT_HMAC_KEY
outpost audit export --type 'dlp.*' --from 2026-03-01 --to 2026-03-15 \
  --verify --hmac-key $AUDIT_HMAC_KEY

outpost audit verify

Verify HMAC chain integrity of the audit buffer.

outpost audit verify --hmac-key $AUDIT_HMAC_KEY
outpost audit verify --hmac-key $AUDIT_HMAC_KEY --json

outpost runbook

Generate a markdown operations runbook from current configuration covering config summary, certificates, circuit breakers, DLP pipeline, heartbeat, and disk usage.

outpost runbook
outpost runbook --json --output runbook.md

---

Configuration hot-reload

Reloadable keys

The following configuration keys can be changed without restarting the container — apply via SIGHUP or outpost config reload:

log_level, dlp_enabled, dlp_ner_enabled, dlp_deberta_enabled, credint_enabled, budget_enforcement_enabled, debug, rate_limit_requests_per_minute, rate_limit_burst, cloud_heartbeat_interval, rate_limit_scan_per_minute, rate_limit_admin_per_minute, log_export_enabled, log_export_format, pushgateway_enabled, pushgateway_interval_seconds, ip_allowlist_enabled, ip_allowlist_cidrs, ip_allowlist_admin_exempt, cert_auto_rotate, cert_rotation_threshold_days, policy_cache_enabled, policy_cache_ttl_seconds

Restart-required keys

These keys require a full container restart to take effect:

outpost_id, org_id, platform_management_url, outpost_cert_path, outpost_key_path, outpost_ca_path, admin_port, outpost_api_key

Hot-reload procedure

1. Edit configuration

   Modify the relevant keys in your .env file or config file.

2. Run config diff

   Preview what will change:

   curl -s -X POST http://127.0.0.1:8301/admin/config/diff \
     -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY" \
     -H "Content-Type: application/json" \
     -d '{"proposed_config": {"LOG_LEVEL": "debug"}}' | jq .

3. Check for breaking changes

   If the diff returns "has_breaking_changes": true, do not proceed with hot-reload. Schedule a maintenance window and perform a full container restart.

4. Apply reload

   Use SIGHUP or the CLI:

   # Via signal
   docker kill --signal=SIGHUP $(docker compose ps -q outpost)
   
   # Via CLI
   outpost config reload --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY
   
   # Via admin API
   curl -s -X POST http://127.0.0.1:8301/admin/api/config/reload \
     -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY"

5. Verify health

   curl -s http://127.0.0.1:8301/admin/api/health | jq '{status, version}'

6. Confirm reload in logs

   Look for the config_reloaded structured log event:

   docker compose logs --tail=50 outpost | jq 'select(.event == "config_reloaded")'

Dry-run validation

Preview the effect of a reload without applying changes:

outpost config reload --admin-key $OUTPOST_EMERGENCY_ADMIN_KEY --dry-run

Atomic rollback

If the reload fails validation, all changes are atomically rolled back — the running configuration remains unchanged. Metrics recorded: outpost_config_reloads_by_result_total{result="success|rejected|failed"} and outpost_config_reload_duration_seconds.

Config backup and restore

# Create a named backup
curl -s -X POST http://127.0.0.1:8301/admin/api/config/backup \
  -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY"

# List backups
curl -s http://127.0.0.1:8301/admin/api/config/backup/list \
  -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY"

# Restore from backup
curl -s -X POST http://127.0.0.1:8301/admin/api/config/restore \
  -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"backup_name": "pre-dlp-change"}'

Config file watcher

Outpost watches OUTPOST_CONFIG_FILE for mtime changes via polling (default interval: 30 seconds). On change, the file is re-parsed and reload_config() is called automatically. YAML nested keys are flattened to uppercase env var names: dlp.enabled: true → DLP_ENABLED=true.

---

Monitoring

Prometheus scrape endpoint

Prometheus metrics are exposed on the admin port (8301):

GET /metrics
Host: <outpost-host>:8301

Add to your prometheus.yml:

scrape_configs:
  - job_name: arbitex-outpost
    static_configs:
      - targets:
          - outpost-host:8301
    metrics_path: /metrics
    scheme: http

Caution

The metrics endpoint is on port 8301 (admin), which should be bound to loopback only. If you need to scrape from a remote Prometheus instance, use a local reverse proxy or Pushgateway.

Metrics reference

Request metrics:

Metric	Type	Labels	Description
outpost_requests_total	Counter	status_code	Total HTTP requests processed
outpost_request_duration_seconds	Histogram	—	HTTP request duration
outpost_active_connections	Gauge	—	Current active proxy connections

DLP metrics:

Metric	Type	Labels	Description
outpost_dlp_scans_total	Counter	tier, action	DLP scans by tier and resulting action
outpost_dlp_scan_latency_seconds	Histogram	tier	DLP scan latency per tier (OTel)
outpost_dlp_stage_duration_seconds	Histogram	tier	Per-tier DLP processing duration

Policy metrics:

Metric	Type	Labels	Description
outpost_policy_evaluations_total	Counter	result	Policy evaluations by result
outpost_policy_evaluations_by_action_total	Counter	action, direction	Policy evaluations by action type and direction
outpost_policy_sync_duration_seconds	Histogram	success	Policy sync duration (OTel)
outpost_policy_sync_age_seconds	Gauge	—	Seconds since last successful policy sync (OTel)

Heartbeat metrics:

Metric	Type	Labels	Description
outpost_heartbeat_success	Gauge	—	1=last heartbeat ok, 0=failed
outpost_heartbeat_rtt_seconds	Histogram	—	Heartbeat round-trip latency
outpost_heartbeat_latency_seconds	Histogram	success	Heartbeat latency (OTel)
outpost_heartbeat_consecutive_failures	Gauge	—	Consecutive heartbeat failures

Circuit breaker metrics:

Metric	Type	Labels	Description
outpost_circuit_breaker_state	Gauge	breaker	0=closed, 1=open, 2=half_open
outpost_circuit_breaker_trips_total	Counter	breaker	Times a breaker has tripped to OPEN
outpost_circuit_breaker_open	Gauge	breaker	1=OPEN, 0=CLOSED/HALF_OPEN (OTel)

Audit and buffer metrics:

Metric	Type	Labels	Description
outpost_audit_buffer_size	Gauge	—	Number of entries in local audit buffer (OTel)
outpost_audit_queue_depth	Gauge	—	Pending events in local SQLite queue (OTel)
outpost_siem_delivery_latency_seconds	Histogram	sink_type	SIEM event delivery latency (OTel)

Config reload metrics:

Metric	Type	Labels	Description
outpost_config_reloads_total	Counter	—	Total config hot-reload operations
outpost_config_reloads_by_result_total	Counter	result	Reloads by result: success, rejected, failed
outpost_config_reload_duration_seconds	Histogram	—	Config reload duration

Infrastructure metrics:

Metric	Type	Labels	Description
outpost_certificate_expiry_seconds	Gauge	cert_type	Seconds until cert expiry
outpost_tls_handshake_failures_total	Counter	reason	TLS handshake failures by reason
outpost_disk_usage_bytes	Gauge	path, type	Disk usage in bytes
outpost_log_rotation_events_total	Counter	result	Log rotation events by result
outpost_shutdown_state	Gauge	—	0=running, 1=draining, 2=stopped

Air-gap metrics:

Metric	Type	Labels	Description
outpost_airgap_last_sync_timestamp	Gauge	—	Unix timestamp of last policy sync
outpost_airgap_pending_updates	Gauge	—	Pending software updates
outpost_airgap_bundle_age_seconds	Gauge	—	Age of current policy bundle

Budget and degradation (OTel):

Metric	Type	Labels	Description
outpost_budget_utilization_ratio	Gauge	—	Budget utilization [0.0–1.0]
outpost_degradation_mode	Gauge	—	1=degraded, 0=normal

Pushgateway (optional)

For environments where the Prometheus server cannot scrape the admin port directly:

PUSHGATEWAY_ENABLED=true
PUSHGATEWAY_URL=http://pushgateway:9091
PUSHGATEWAY_JOB=outpost
PUSHGATEWAY_INTERVAL_SECONDS=60

Status: GET /admin/api/pushgateway/status

OpenTelemetry

Outpost emits OTel traces via auto-instrumentation for FastAPI and httpx:

OTEL_ENABLED=true
OTEL_EXPORTER_ENDPOINT=http://otel-collector:4317
OTEL_SERVICE_NAME=arbitex-outpost

Status: GET /admin/api/otel/status

Metrics summary endpoint

GET /admin/api/metrics-summary returns a structured JSON snapshot of key metric values (not raw Prometheus text format).

---

Air-gap mode

Enable air-gap mode by setting OUTPOST_AIRGAP=true. This disables all outbound network calls and switches to filesystem-based workflows.

Path defaults

Variable	Default	Description
AIRGAP_POLICY_PATH	/opt/arbitex/policies	Directory for policy bundles
AIRGAP_MODEL_PATH	/opt/arbitex/models	Directory for DLP model files

Normal vs air-gap behavior

Feature	Normal mode	Air-gap mode
Policy sync	Pull from platform on interval	Load from AIRGAP_POLICY_PATH at startup
Heartbeat	POST to platform every 60 s	Disabled (no network call)
Cert rotation	Auto-download from platform	Manual cert replacement + restart
GeoIP	Download on start or explicit path	Explicit path or bundled only
CredInt bloom	CDN refresh on schedule	Manual .bf file placement; reload via SIGHUP
DeBERTa model	From DEBERTA_MODEL_PATH	Must be at AIRGAP_MODEL_PATH
Config checks	platform_reach and heartbeat checks run	These checks skipped
k-anonymity	Available if CREDINT_KANON_ENABLED=true	Disabled (requires network)

Startup sequence differences

In air-gap mode, the startup sequence skips all platform connectivity steps:

1. Load config and validate required env vars (PLATFORM_MANAGEMENT_URL not required)
2. Validate mTLS certs from filesystem
3. Load policy bundle from AIRGAP_POLICY_PATH (fail if missing)
4. Initialize DLP pipeline from local models
5. Start proxy and admin listeners

CredInt bloom filter provisioning (air-gap)

In air-gap mode, set CREDINT_BLOOM_PATH to the bloom filter file and leave CREDINT_CDN_URL empty. To update the bloom filter:

1. Download a fresh .bf file on an internet-connected workstation
2. Transfer to the air-gap host
3. Replace the file at CREDINT_BLOOM_PATH
4. Reload config: outpost config reload or send SIGHUP

Air-gap monitoring

Air-gap metrics are populated for offline monitoring:

• outpost_airgap_last_sync_timestamp — Unix timestamp of policy load
• outpost_airgap_pending_updates — staged bundles awaiting apply
• outpost_airgap_bundle_age_seconds — age of current policy bundle

Air-gap config: GET /admin/api/airgap-config

---

Log rotation configuration

Outpost writes structured JSON logs to stdout by default. In environments where logs must be retained on disk (air-gap, compliance hold, local SIEM), the built-in rotating file handler can be enabled.

Environment variables

Variable	Type	Default	Description
LOG_ROTATION_ENABLED	bool	false	Enable on-disk log rotation via RotatingFileHandler
LOG_MAX_BYTES	int	104857600	Maximum log file size in bytes before rotation (100 MB)
LOG_BACKUP_COUNT	int	5	Number of rotated backup files to retain

How rotation works

When LOG_ROTATION_ENABLED=true, Outpost writes structured logs using Python’s RotatingFileHandler. When the active log file reaches LOG_MAX_BYTES, the handler renames it to .log.1, shifts older files, and starts a fresh log. Files beyond LOG_BACKUP_COUNT are deleted automatically.

Docker volume mount

services:
  outpost:
    volumes:
      - outpost-logs:/var/log/arbitex
    environment:
      - LOG_ROTATION_ENABLED=true
      - LOG_MAX_BYTES=104857600
      - LOG_BACKUP_COUNT=5

volumes:
  outpost-logs:

External log shipper integration

Mount the log volume read-only into a sidecar shipper:

services:
  outpost:
    volumes:
      - outpost-logs:/var/log/arbitex
  fluentd:
    image: fluent/fluentd:v1.16
    volumes:
      - outpost-logs:/var/log/arbitex:ro

volumes:
  outpost-logs:

Configure Fluentd’s tail input plugin to follow /var/log/arbitex/outpost.log* with follow_inodes true. See SIEM integration guide for Splunk, Elastic, and Sentinel forwarding patterns.

---

Disk usage alerting

Outpost includes a built-in disk monitor that emits structured log events and Prometheus metrics when disk usage crosses thresholds.

Alert log format

{
  "event": "disk_usage_alert",
  "level": "warning",
  "path": "/",
  "usage_pct": 76.3,
  "total_bytes": 107374182400,
  "used_bytes": 81938505728,
  "free_bytes": 25435676672,
  "threshold": "warn"
}

When usage crosses the critical threshold (default 90%), the same event is emitted with "level": "error" and "threshold": "critical".

Caution

A threshold=critical disk alert means the monitored filesystem is approaching full. At 100% utilisation, Outpost cannot write audit events or rotate log files, which causes requests to be rejected under fail-closed policy. Treat critical disk alerts as P1.

The gauge outpost_disk_usage_bytes{path, type} is available for Prometheus scraping.

---

Troubleshooting

HTTP error codes

Code	Meaning	Common cause
401 Unauthorized	Missing or invalid Authorization header	Expired API key or missing mTLS cert
403 Forbidden	Request blocked by policy	Policy action resolved to block
413 Content Too Large	Request body exceeds configured limit	MAX_REQUEST_BODY_MB or REQUEST_MAX_BODY_BYTES
429 Too Many Requests	Rate limit or brute-force protection triggered	Exceeded rate limit or admin API failure threshold
503 Service Unavailable	Outpost not ready	Critical component not yet initialized
500 Internal Server Error	Unexpected internal error	Check logs for stack trace

Admin API brute-force protection

After 5 failed authentication attempts within a 15-minute window, the admin API returns 429 Too Many Requests for all subsequent requests until the window expires.

Startup hard blockers

The following conditions cause an immediate fatal exit at startup:

• AUDIT_HMAC_KEY is empty
• POLICY_HMAC_KEY is empty and INSECURE_SKIP_HMAC is not set to true

Common issues

Symptom	Likely cause	Resolution
/ready returns 503	Critical component not initialized	Run outpost validate-config; check logs for init errors
policy_sync error	Bundle age > 3600 s or unreachable platform	Check PLATFORM_MANAGEMENT_URL connectivity; check cert validity
Certificate expiry warning	mTLS cert nearing expiry	Rotate cert; ensure CERT_AUTO_ROTATE=true
NER disabled	DLP_NER_ENABLED=false or spaCy model missing	Verify DLP_NER_MODEL installed; re-enable NER
DeBERTa shows as disabled	DLP_DEBERTA_ENABLED=true but model missing	Set DEBERTA_MODEL_PATH to a valid ONNX model file
GeoIP disabled	No MMDB file found	Set GEOIP_MMDB_PATH or enable GEOIP_DOWNLOAD_ON_START
Update state verify_failed	Bundle signature mismatch	Verify bundle provenance; confirm Ed25519 public key
Air-gap policy not found	No bundle at AIRGAP_POLICY_PATH	Place a signed policy_bundle.json at the configured path
Circuit breaker OPEN	Repeated failures to reach platform	Investigate network path; check mTLS certs; wait for recovery
Audit buffer growing	audit_sync breaker OPEN	Check platform connectivity; verify mTLS cert
Config reload rejected	Restart-required key changed	Full container restart required for identity, cert, or port changes
SIGHUP did not reload certs	TLS session using old cert	Full container restart if SIGHUP insufficient

Diagnostic commands

# Full diagnostic report
outpost diagnostics --json --output-file diagnostics.json

# Validate all config checks
outpost validate-config --json

# Export current health as JSON
outpost health export --output health.json

# View recent container logs
docker compose logs --tail=200 outpost

# Query admin API health
curl -s -H "Authorization: Bearer $OUTPOST_EMERGENCY_ADMIN_KEY" \
  http://127.0.0.1:8301/admin/api/health | jq .

# Pull current Prometheus metrics
curl -s http://localhost:8301/metrics | grep outpost_heartbeat

# Check cert expiry
outpost certs status

# Verify audit chain
outpost audit verify --hmac-key $AUDIT_HMAC_KEY --json

# Generate ops runbook from current config
outpost runbook --output runbook.md

Diagnostic log patterns

Policy sync failures

level=error msg="policy_sync failed" error="connection refused" attempts=3
level=error msg="policy_bundle_verify failed" error="HMAC mismatch" bundle_version="v20260314-001"
level=warning msg="circuit breaker opened" breaker="policy_sync" consecutive_failures=5
level=warning msg="policy bundle age exceeded threshold" age_seconds=360 threshold_seconds=300

Certificate issues

level=warning msg="cert_expiry_soon" days_remaining=14 expiry="2026-09-01T00:00:00Z"
level=info msg="cert_renewal_started" current_expiry="2026-09-01T00:00:00Z"
level=info msg="cert_renewal_complete" new_expiry="2027-03-01T00:00:00Z"
level=error msg="cert_renewal_failed" error="CSR submission rejected: 403 Forbidden"
level=info msg="tls_config_reloaded" cert_path="/app/certs/outpost.pem"

Audit buffer

level=debug msg="audit_event_buffered" event_id="evt-20260314-001823" buffer_size=42
level=info msg="audit_batch_uploaded" count=500 duration_ms=213
level=error msg="audit_batch_upload_failed" error="connection timeout" retry_in_seconds=30
level=warning msg="circuit breaker opened" breaker="audit_sync" pending_events=12500

Heartbeat

level=info msg="heartbeat_sent" version="0.1.0" rtt_ms=45
level=error msg="heartbeat_failed" error="connection refused" attempt=2
level=warning msg="version_mismatch" running="0.1.0" latest="1.5.0"
level=info msg="shutdown_heartbeat_sent" shutting_down=true

Config reload

level=info msg="config_reloaded" changed_keys=["LOG_LEVEL","DLP_ENABLED"] has_breaking=false
level=warning msg="config_reload_rejected" reason="validation failed" errors=["invalid_rate_limit"]
level=error msg="config_reload_failed" error="atomic rollback triggered"

Log event reference

Scenario	event field	Key fields to check
Startup	startup_complete	version, listen_port, dlp_enabled
DLP detection	dlp_detection	entity_type, confidence, action, tier
Circuit breaker trip	circuit_breaker_opened	provider, failure_count, threshold
Config reload	config_reloaded	changed_keys, has_breaking
Disk alert	disk_usage_alert	usage_pct, threshold, path
Auth failure	auth_failed	reason, client_ip, token_type

Filtering logs with jq

All Outpost log output is newline-delimited JSON when LOG_FORMAT=json. Use jq with the select() function:

# Stream live events for a specific type
docker compose logs -f outpost | jq -c 'select(.event == "dlp_detection")'

# Show all auth failures in the last 200 lines
docker compose logs --tail=200 outpost | jq 'select(.event == "auth_failed")'

# Count DLP detections by entity type
docker compose logs --tail=1000 outpost | \
  jq -r 'select(.event == "dlp_detection") | .entity_type' | sort | uniq -c | sort -rn