Outpost Reliability Features

Outpost Reliability Features

Arbitex Outpost is designed to keep serving AI requests even when connectivity to the Arbitex platform is degraded or lost. This guide covers the reliability features shipped in outpost sprints 0029 through 0032: circuit breakers, graceful degradation, local audit queue, certificate rotation, air-gap operation, and the admin CLI.

---

Circuit Breaker

The circuit breaker prevents an offline or overloaded platform from causing request storms. It is applied to every outbound platform API call: policy sync, heartbeat, and audit log sync.

States

CLOSED ──(failure threshold)──▶ OPEN ──(recovery timeout)──▶ HALF_OPEN ──(probe success)──▶ CLOSED
                                  ▲                                           │
                                  └──────────────(probe failure)──────────────┘

State	Behavior
CLOSED	Normal operation — all calls pass through
OPEN	All calls fail immediately with CircuitOpenError (no network attempt)
HALF_OPEN	One probe call is allowed through; success → CLOSED, failure → OPEN

Defaults

Parameter	Default	Description
failure_threshold	5	Consecutive failures to trip CLOSED → OPEN
recovery_timeout_seconds	60	Seconds in OPEN before entering HALF_OPEN
half_open_max_calls	1	Probe calls allowed in HALF_OPEN before decision

State transitions are logged at INFO level. All circuit breaker instances are tracked in CircuitBreakerRegistry (singleton).

Scope

Three platform API call sites use circuit breakers:

Client	What it protects
PolicySyncClient	Policy rules download from platform
HeartbeatSender	Outpost → platform keepalive
AuditSyncWorker	Audit event upload to platform

Each has its own named circuit breaker. If only audit sync is failing, the audit breaker opens independently — policy sync and heartbeats continue unaffected.

---

Graceful Degradation

When any circuit breaker is open, the outpost enters degradation mode. In this mode:

• Requests continue to be served using cached policy rules.
• Audit events are queued locally to SQLite instead of being uploaded immediately.
• Policy sync returns the last successfully downloaded policy instead of failing.

The DegradationManager (outpost/degradation.py) monitors circuit breaker state and transitions:

• Enter: Logged at WARNING level with timestamp.
• Exit: Logged at INFO level with degradation duration.

The admin health endpoint reports degradation state:

{
  "status": "degraded",
  "degradation_mode": true,
  "degraded_since": "2026-03-12T14:23:01Z"
}

---

Local Audit Queue

When the outpost is degraded, audit events are written to a SQLite queue (audit_queue/audit_queue.db) instead of being sent to the platform.

Queue behavior

Operation	Description
enqueue	Append event to SQLite; thread-safe via threading.Lock
dequeue_batch	Fetch N oldest events for sync attempt
mark_synced	Remove successfully uploaded events
increment_retry	Increment retry counter on upload failure
purge	Delete all queued events (requires admin confirmation)

Events are retried in FIFO order on recovery. The queue is persistent across outpost restarts — events survive process crashes or rolling restarts.

Sync-on-recovery

When circuit breakers return to CLOSED (the platform becomes reachable again), AuditSyncWorker._flush_local_queue() automatically drains queued events to the platform before resuming normal sync operation.

Configuration

AUDIT_QUEUE_DB_PATH=audit_queue/audit_queue.db   # Path to SQLite queue file

The default path is relative to the outpost working directory. Use an absolute path for production deployments.

Admin endpoints

Endpoint	Method	Description
/admin/audit-queue/status	GET	Queue depth, oldest event timestamp, disk usage
/admin/audit-queue/flush	POST	Trigger immediate sync attempt (409 if queue empty)
/admin/audit-queue/purge	DELETE	Delete all queued events (requires X-Purge-Confirm: yes header)

---

Certificate Rotation

The outpost supports automatic certificate bundle download from the platform at startup, as well as manual cert rotation.

Auto-download at startup

When CERT_BUNDLE_AUTO_DOWNLOAD=true, the outpost calls CertBundleDownloader.download_and_install() at startup:

1. Fetches the cert bundle from the platform using mTLS.
2. Verifies the RSA or EC certificate chain (leaf → intermediate → CA).
3. Writes leaf.pem, intermediate.pem, and ca.pem to the configured cert directory.
4. Fails open on any error — if download fails, the outpost starts with existing certs.

Rotation lifecycle

Old cert bundle            New cert bundle
─────────────────────────────────────────────────────
Startup: download+install   File overwrite (atomic)
Chain verify: leaf→int→CA   Verified before write
On failure:                 Old certs preserved

Configuration

CERT_BUNDLE_AUTO_DOWNLOAD=false   # Enable cert bundle download at startup (default: false)
CERT_DIR=/path/to/certs           # Directory for leaf.pem / intermediate.pem / ca.pem

Validation

The outpost validate-config CLI checks cert files at startup:

Condition	Result
Certificate expired	[FAIL] — exit code 1
Certificate expires in < 30 days	[WARN] — continues
Certificate valid	[PASS]
Missing cert file	No crash — logged

---

Air-Gap Mode Operation

Air-gap mode allows the outpost to run in isolated networks with no connectivity to the Arbitex platform. All platform-dependent sync operations are disabled; the outpost runs entirely from a local bundle.

What is disabled in air-gap mode

Feature	Behavior
Policy sync	Skipped — loads from local AirGapLoader bundle
Heartbeat	Skipped
Audit sync to platform	Skipped — events remain in local queue
Cert bundle auto-download	Skipped (guard prevents attempt)

Bundle loading

Air-gap bundles are loaded by AirGapLoader. A missing bundle is handled gracefully — the outpost logs a warning and continues with last-known-good policy or an empty policy.

Enabling air-gap mode

AIR_GAP=true   # Enable air-gap operation

When AIR_GAP=true:

• All HTTP calls to platform URLs are suppressed at the call site (guard checks before attempt, not at network layer).
• The circuit breaker is bypassed — there is no “circuit open” in air-gap mode since no calls are attempted.

---

Admin CLI

The outpost CLI provides operational commands for checking status and validating configuration without restarting the process.

Installation

The CLI is installed as an entry point via pyproject.toml:

outpost status
outpost validate-config
# or
python -m outpost.cli status
python -m outpost.cli validate-config

outpost status

Queries the outpost admin API and displays current operational state.

outpost status [--json] [--admin-port PORT] [--admin-key KEY]

Human-readable output:

Outpost Status
  Status:          healthy
  Circuit open:    false
  Degradation:     false
  Policy age:      42s
  Queue depth:     0
  Audit sync:      2026-03-12T14:20:00Z

Circuit Breakers:
  policy_sync      CLOSED
  heartbeat        CLOSED
  audit_sync       CLOSED

JSON output (--json):

{
  "status": "healthy",
  "circuit_breakers": {
    "policy_sync": "CLOSED",
    "heartbeat": "CLOSED",
    "audit_sync": "CLOSED"
  },
  "degradation_mode": false,
  "audit_queue_depth": 0,
  "last_sync": "2026-03-12T14:20:00Z"
}

Exit codes:

Code	Condition
0	Healthy
1	Connection error, auth failure (401), server error (500), or timeout

outpost validate-config

Runs all startup configuration checks and reports results. Does not require the outpost to be running.

outpost validate-config [--json]

Output:

[PASS] mTLS client certificate: valid (expires 2027-01-15)
[PASS] mTLS CA certificate: valid (expires 2028-06-30)
[WARN] mTLS intermediate: expires in 22 days (2026-04-03)
[PASS] GeoIP MMDB: valid (MaxMind magic bytes present)
[PASS] Platform URL: reachable
[FAIL] Audit queue directory: not writable

Checks performed:

Check	Pass condition	Fail condition
mTLS cert expiry	Valid and > 30 days	Expired
mTLS cert expiry (warning)	Valid and > 30 days	< 30 days remaining
GeoIP MMDB	Magic bytes present (\xab\xcd\xef)	Absent (WARN)
Platform URL reachability	HTTP 200 or 401	Connection error
Audit queue directory	Writable	Not writable

Exit code 1 if any [FAIL] check; exit 0 otherwise (warnings do not affect exit code).

---

Prometheus Metrics

The outpost exposes the following reliability-related metrics at /metrics:

Metric	Type	Labels	Description
outpost_circuit_breaker_open	Gauge	breaker=<name>	1 if circuit open, 0 if closed
outpost_degradation_mode	Gauge	—	1 if degraded, 0 if normal
outpost_audit_queue_depth	Gauge	—	Number of events in local SQLite queue
outpost_requests_total	Counter	status=success|error	Total requests handled
outpost_policy_sync_age_seconds	Gauge	—	Seconds since last successful policy sync

These metrics are refreshed on every GET /metrics request.

---

Related Documentation

• Outpost Deployment — installation and initial configuration
• Outpost JWT Validation — authentication setup
• Distributed Tracing — OTel configuration for outpost telemetry
• Audit Log Management — understanding audit events