Network Resilience Guide

Arbitex Outpost includes built-in network resilience features that monitor connection quality, adapt sync behavior to available bandwidth, and gracefully degrade when the platform is unreachable. This guide covers configuration, threshold tuning, and operational behavior for each resilience layer.

---

Architecture overview

The resilience stack has four layers, each building on the one below:

Layer	Component	Purpose
1	Circuit breakers	Fail-fast on repeated failures, prevent cascading overload
2	Connection health monitor	Track latency, packet loss, and jitter to the platform
3	Bandwidth adapter	Adjust audit sync batch sizes based on measured throughput
4	Degradation manager	Coordinate system-wide behavior when connectivity is lost

---

Connection health monitoring

The connection health monitor sends periodic probes to the platform and computes rolling metrics over a sliding window.

How it works

Every 30 seconds (default), the monitor sends an HTTP HEAD request to {PLATFORM_MANAGEMENT_URL}/health. Each probe measures:

• Whether the request succeeded or failed
• Round-trip latency in milliseconds

Results are stored in a sliding window (default: 60 probes = 30 minutes of history). After each probe, the monitor recomputes rolling metrics and updates Prometheus gauges.

Configuration

Variable	Default	Description
CONNECTION_HEALTH_WINDOW_SIZE	60	Number of probes in the sliding window
CONNECTION_HEALTH_LATENCY_WARN_MS	500	Latency threshold for degraded status (ms)
CONNECTION_HEALTH_LATENCY_CRIT_MS	2000	Latency threshold for critical status (ms)
CONNECTION_HEALTH_LOSS_WARN_PCT	5.0	Packet loss threshold for degraded status (%)
CONNECTION_HEALTH_LOSS_CRIT_PCT	20.0	Packet loss threshold for critical status (%)

Health states

The monitor evaluates thresholds in priority order — critical conditions are checked first:

Condition	Status	Prometheus value
Packet loss ≥ 20%	critical	3
Average latency ≥ 2000 ms	critical	3
Packet loss ≥ 5%	degraded	2
Average latency ≥ 500 ms	degraded	2
All probes within thresholds	healthy	1
No probes in window yet	unknown	0

Metrics computed

Metric	Description
avg_latency_ms	Arithmetic mean of successful probe latencies
p95_latency_ms	95th percentile latency
p99_latency_ms	99th percentile latency
packet_loss_pct	(failed probes / total probes) × 100
jitter_ms	Standard deviation of successful probe latencies
probe_count	Total probes in the current window
success_count	Successful probes in the current window

Checking connection health

GET /admin/api/connection-health
Authorization: Bearer <admin-token>

{
  "status": "healthy",
  "total_probes": 60,
  "avg_latency_ms": 42.3,
  "p95_latency_ms": 89.1,
  "p99_latency_ms": 124.7,
  "packet_loss_pct": 0.0,
  "jitter_ms": 12.4,
  "probe_count": 60,
  "success_count": 60
}

Viewing probe history

GET /admin/api/connection-health/history
Authorization: Bearer <admin-token>

Returns the raw sliding window as a JSON array:

[
  { "timestamp": 1710542400.0, "success": true, "latency_ms": 38.2 },
  { "timestamp": 1710542430.0, "success": true, "latency_ms": 45.1 },
  { "timestamp": 1710542460.0, "success": false, "latency_ms": null }
]

Tuning thresholds

Low-latency environments (same datacenter or region): tighten the warning threshold to catch issues earlier:

CONNECTION_HEALTH_LATENCY_WARN_MS=200
CONNECTION_HEALTH_LATENCY_CRIT_MS=1000

High-latency environments (cross-region, satellite links): relax thresholds to avoid false alarms:

CONNECTION_HEALTH_LATENCY_WARN_MS=1000
CONNECTION_HEALTH_LATENCY_CRIT_MS=5000
CONNECTION_HEALTH_LOSS_WARN_PCT=10.0
CONNECTION_HEALTH_LOSS_CRIT_PCT=30.0

---

Bandwidth-aware sync tuning

The bandwidth adapter dynamically adjusts audit sync batch sizes based on measured network throughput. Larger batches are more efficient when bandwidth is plentiful; smaller batches prevent timeouts on constrained links.

4-tier batch mapping

Throughput	Tier	Batch size
≥ 1 MB/s (1,048,576 bps)	high	100
≥ 256 KB/s (262,144 bps)	medium	50
≥ 64 KB/s (65,536 bps)	low	25
< 64 KB/s	minimal	10

When no throughput data is available (empty measurement window), the adapter defaults to batch size 50 (medium tier).

How throughput is measured

The audit sync worker reports each transfer’s payload size and wall-clock duration after every successful sync. The adapter maintains a sliding window of the last 20 observations and computes the arithmetic mean throughput in bytes per second.

Connection health integration

When the connection health monitor is active, the bandwidth adapter applies a health-based cap on top of the throughput-derived batch size:

Connection status	Batch cap
critical	10 (forces minimal tier)
degraded	25 (caps at low tier)
healthy / unknown	No cap

The final batch size is: min(throughput_batch, health_cap), clamped to [batch_min, batch_max].

Configuration

Variable	Default	Description
AUDIT_SYNC_BATCH_MIN	10	Minimum batch size floor
AUDIT_SYNC_BATCH_MAX	100	Maximum batch size ceiling

Checking bandwidth status

GET /admin/api/bandwidth
Authorization: Bearer <admin-token>

{
  "enabled": true,
  "throughput_bps": 524288.0,
  "window_size": 15,
  "window_capacity": 20,
  "recommended_batch_size": 50,
  "tier_name": "medium",
  "batch_min": 10,
  "batch_max": 100,
  "connection_health_status": "healthy"
}

---

Circuit breakers

Circuit breakers prevent the Outpost from overwhelming a failing platform connection with repeated requests. Each critical service has its own circuit breaker.

Registered circuit breakers

Name	Protects
policy_sync	Policy bundle sync from platform
heartbeat	Periodic heartbeat to platform
audit_sync	Audit event sync to platform
connection_health	Health probe to platform

State machine

CLOSED ──(threshold consecutive failures)──→ OPEN
   ↑                                           │
   │                                    (recovery timeout)
   │                                           ↓
   └──────(probe succeeds)──────────── HALF_OPEN
                                          │
                              (probe fails)──→ OPEN

State	Behavior
CLOSED	Requests flow normally. Consecutive failures are counted.
OPEN	All requests fail immediately with CircuitOpenError. No network calls made.
HALF_OPEN	A limited number of probe requests are allowed through. Success → CLOSED. Failure → OPEN.

Configuration

Variable	Default	Description
CIRCUIT_BREAKER_FAILURE_THRESHOLD	5	Consecutive failures to trip the breaker
CIRCUIT_BREAKER_RECOVERY_TIMEOUT	30 seconds	Time in OPEN state before trying HALF_OPEN
CIRCUIT_BREAKER_HALF_OPEN_MAX_REQUESTS	3	Max probe requests allowed in HALF_OPEN state

Trip conditions

• CLOSED → OPEN: When consecutive failure count reaches failure_threshold (default: 5)
• HALF_OPEN → OPEN: Any failure or timeout during a probe request. Recovery timer resets.
• HALF_OPEN slot exhaustion: If half_open_max_calls probes are already in flight, additional calls raise CircuitOpenError without counting as failures.

Recovery

• OPEN → HALF_OPEN: After recovery_timeout_seconds elapses since the circuit opened
• HALF_OPEN → CLOSED: When a probe request succeeds. Failure counter resets to zero.

HALF_OPEN probes have a 5-second timeout. If the probe times out, the circuit reopens.

Monitoring circuit breakers

Circuit breaker states are included in the metrics endpoint:

GET /admin/api/metrics
Authorization: Bearer <admin-token>

The response includes a circuit_breakers array:

{
  "circuit_breakers": [
    { "name": "policy_sync", "state": "closed", "status": "green" },
    { "name": "heartbeat", "state": "closed", "status": "green" },
    { "name": "audit_sync", "state": "open", "status": "red" },
    { "name": "connection_health", "state": "half_open", "status": "yellow" }
  ]
}

The summary flag circuit_breaker_open in GET /admin/api/status is true if any circuit breaker is open.

---

Degraded mode behavior

When circuit breakers trip, the degradation manager coordinates system-wide behavior across three levels.

Degradation levels

Level	Condition	Behavior
NORMAL (0)	All circuits closed	Full operation — syncs, heartbeats, and health probes run normally
DEGRADED (1)	Some circuits open	Stale policy cache is used. Audit events buffer to local SQLite queue. Heartbeat retries continue.
ISOLATED (2)	All circuits open, or degraded for > 5 minutes	Full local operation. All syncs suspended. Audit queue preserves events for later sync.

Escalation

• Any circuit breaker opening → DEGRADED
• All circuit breakers open simultaneously → ISOLATED (immediate)
• Remaining in DEGRADED for longer than the escalation timeout (default: 300 seconds) → ISOLATED
• All circuit breakers close → NORMAL

What continues working in degraded mode

Feature	NORMAL	DEGRADED	ISOLATED
Proxy requests to AI providers	Yes	Yes	Yes
DLP scanning	Yes	Yes (stale policy)	Yes (stale policy)
Policy sync	Yes	Retrying	Suspended
Audit event capture	Yes	Yes (local buffer)	Yes (local buffer)
Audit sync to platform	Yes	Retrying	Suspended
Heartbeat	Yes	Retrying	Suspended
Connection health probes	Yes	Yes	Yes

Tip

Audit events are never dropped in any degradation level. They accumulate in the local SQLite queue (audit_queue.db) and are synced to the platform when connectivity is restored.

Recovery

When the platform becomes reachable again:

1. Circuit breakers transition through HALF_OPEN → CLOSED as probes succeed
2. Degradation level returns to NORMAL
3. Audit sync worker drains the local queue in bandwidth-adapted batches
4. Policy sync fetches the latest policy bundle
5. Heartbeat resumes on its normal interval

---

Operational recommendations

Alerting thresholds

Set up alerts on these Prometheus metrics:

Metric	Alert condition	Severity
outpost_connection_health_status	>= 2 (degraded) for 5 min	Warning
outpost_connection_health_status	== 3 (critical) for 2 min	Critical
outpost_circuit_breaker_state	!= 0 (not closed) for 5 min	Warning
outpost_audit_queue_depth	> 1000	Warning
outpost_audit_queue_depth	> 10000	Critical

Network requirements

Traffic	Protocol	Destination	Notes
Health probes	HTTPS (HEAD)	Platform /health	Every 30 s, plain TLS (not mTLS)
Policy sync	HTTPS	Platform management URL	mTLS, periodic
Audit sync	HTTPS	Platform management URL	mTLS, batch-adaptive
Heartbeat	HTTPS	Platform management URL	mTLS, periodic

Capacity planning for offline periods

During extended outages, audit events accumulate locally. Estimate storage needs:

• Average audit event size: ~2 KB
• At 100 requests/minute: ~12 MB/hour, ~288 MB/day
• The SQLite queue has no hard size limit — plan disk space accordingly

After reconnection, the bandwidth adapter ensures sync does not saturate the link. Recovery time depends on queue depth and available bandwidth.