Air-gap operations guide

This guide covers day-to-day operations for an Arbitex Outpost running in an air-gapped environment. It assumes the outpost was installed using the air-gap deployment guide.

Service management

systemd (recommended)

If the systemd service was installed during setup, use standard systemctl commands. The service name is arbitex-outpost.

# Check service status
sudo systemctl status arbitex-outpost

# Start the service
sudo systemctl start arbitex-outpost

# Stop the service
sudo systemctl stop arbitex-outpost

# Restart the service (required after .env changes or certificate replacement)
sudo systemctl restart arbitex-outpost

# View live logs
journalctl -u arbitex-outpost -f

# View the last 200 log lines
journalctl -u arbitex-outpost -n 200

# View logs since a specific time
journalctl -u arbitex-outpost --since "2026-03-01 00:00:00"

Docker Compose (without systemd)

If the outpost is not running under systemd, manage it directly with Docker Compose from the install directory:

cd /opt/arbitex-outpost

# View running containers
docker compose -f docker-compose.outpost.yml ps

# Start in foreground
docker compose -f docker-compose.outpost.yml up

# Start in background
docker compose -f docker-compose.outpost.yml up -d

# Stop
docker compose -f docker-compose.outpost.yml down

# View live logs
docker compose -f docker-compose.outpost.yml logs -f

# Restart the outpost container
docker compose -f docker-compose.outpost.yml restart

Health probes

# Liveness — returns 200 if the process is running
curl http://localhost:8300/healthz

# Readiness — returns 200 if a policy bundle is loaded; 503 if not
curl http://localhost:8300/readyz

Offline update procedure

Updating the outpost in an air-gapped environment requires transferring a new air-gap package and replacing the running container images. No internet access is required on the target host.

1. Build the new package

On the internet-connected build machine:

bash scripts/make-airgap.sh <NEW_VERSION>

Output: dist/arbitex-outpost-airgap-<NEW_VERSION>.tar.gz and .sha256.

2. Transfer the package

Transfer the new tarball and checksum to the air-gapped host using any available channel (SCP, removable media, internal file transfer):

scp dist/arbitex-outpost-airgap-<NEW_VERSION>.tar.gz \
    dist/arbitex-outpost-airgap-<NEW_VERSION>.tar.gz.sha256 \
    user@target-host:/tmp/

3. Verify and extract

On the target host:

cd /tmp
sha256sum -c arbitex-outpost-airgap-<NEW_VERSION>.tar.gz.sha256
tar -xzf arbitex-outpost-airgap-<NEW_VERSION>.tar.gz

4. Load the new Docker images

cd airgap-<NEW_VERSION>/
docker load < outpost-image-<NEW_VERSION>.tar.gz

Both the CPU and GPU images are loaded from the tarball. No registry pull or internet access occurs.

5. Update docker-compose to reference the new image tag

Edit /opt/arbitex-outpost/docker-compose.outpost.yml and update the image tag from <OLD_VERSION> to <NEW_VERSION>.

Alternatively, if the Compose file references an environment variable for the image tag, update .env:

# In /opt/arbitex-outpost/.env
OUTPOST_IMAGE_TAG=<NEW_VERSION>

6. Restart the outpost

If running under systemd:

sudo systemctl restart arbitex-outpost

With Docker Compose directly:

cd /opt/arbitex-outpost
docker compose -f docker-compose.outpost.yml down
docker compose -f docker-compose.outpost.yml up -d

7. Verify the update

curl http://localhost:8300/healthz
curl http://localhost:8300/readyz

Check logs for the new version string at startup:

journalctl -u arbitex-outpost -n 50

Rollback

If the new version fails health checks, reload the previous image (if still in the Docker image cache) and revert the image tag reference:

# Check what images are available locally
docker images | grep arbitex/outpost

# Revert docker-compose image tag to the previous version, then restart
sudo systemctl restart arbitex-outpost

Certificate management

Arbitex Outpost requires three mTLS certificate files for management plane connectivity. In internet-connected deployments, the CertRotationClient handles automatic renewal. In air-gapped environments, certificate renewal is a manual procedure.

Certificate locations

Default paths (relative to install directory, e.g. /opt/arbitex-outpost/):

File	Variable	Default path
Client certificate	`OUTPOST_CERT_PATH`	`certs/outpost.pem`
Private key	`OUTPOST_KEY_PATH`	`certs/outpost.key`
CA certificate	`OUTPOST_CA_PATH`	`certs/ca.pem`

Checking certificate expiry

openssl x509 -in /opt/arbitex-outpost/certs/outpost.pem -noout -enddate

The Outpost logs a warning when the certificate is within 30 days of expiry.

Manual certificate renewal

Contact Arbitex support or log in to cloud.arbitex.ai to request a renewed certificate bundle for your outpost ID.
Download the new outpost.pem, outpost.key, and (if the CA changed) ca.pem.
Transfer the new files to the air-gapped host.
Replace the certificate files:

cp new-outpost.pem /opt/arbitex-outpost/certs/outpost.pem
cp new-outpost.key /opt/arbitex-outpost/certs/outpost.key
# Only replace ca.pem if the CA certificate itself changed
# cp new-ca.pem /opt/arbitex-outpost/certs/ca.pem

Restart the outpost to reload the certificates:

sudo systemctl restart arbitex-outpost

Verify TLS connectivity by checking that the heartbeat and policy sync succeed in the logs:

journalctl -u arbitex-outpost -f | grep -E "policy_sync|heartbeat|cert"

Self-signed certificates (management plane unreachable)

In air-gap environments where the Outpost cannot reach the management plane at all (fully network-isolated), the outpost operates using the locally cached policy bundle. Certificate expiry in this mode does not prevent the proxy from serving requests — only management plane connectivity is affected. The local audit log, DLP pipeline, and request routing continue to function using the cached bundle.

Policy bundle management

Current bundle status

The active policy bundle is stored at policy_cache/policy_bundle.json (relative to install directory). Inspect the current bundle version:

cat /opt/arbitex-outpost/policy_cache/policy_bundle.json | python3 -m json.tool | grep '"version"'

Updating the policy bundle offline

When the outpost can temporarily reach the management plane (e.g., during a brief network window), the PolicySyncClient automatically fetches the latest bundle. No manual action is required.

If you need to push a specific policy bundle to an outpost that has no management plane connectivity at all:

Export the policy bundle from the cloud portal as a JSON file.
Transfer it to the air-gapped host.
Replace the cached bundle:

cp exported-policy-bundle.json /opt/arbitex-outpost/policy_cache/policy_bundle.json

Restart the outpost to reload the bundle:

sudo systemctl restart arbitex-outpost

Log rotation

Application logs (via systemd journal)

The systemd journal rotates automatically based on /etc/systemd/journald.conf. To limit journal storage for the outpost service:

# Set a size cap on the journal (edit /etc/systemd/journald.conf)
SystemMaxUse=500M

Then restart journald:

sudo systemctl restart systemd-journald

Audit buffer

The audit buffer at audit_buffer/audit.jsonl is managed as a ring buffer by the outpost. The maximum number of entries is set by MAX_AUDIT_BUFFER_ENTRIES (default: 100,000). When the buffer is full, the oldest entries are rotated out automatically.

For long-running air-gap deployments where audit events accumulate without being synced to the management plane, monitor the buffer size periodically:

wc -l /opt/arbitex-outpost/audit_buffer/audit.jsonl
du -sh /opt/arbitex-outpost/audit_buffer/audit.jsonl

If the buffer approaches capacity, extract and archive audit events (see below) before they are rotated out.

SIEM dead-letter file

If SIEM_DIRECT_DEAD_LETTER_PATH is set, failed SIEM deliveries are appended to that file without rotation. Add an OS-level log rotation entry if SIEM forwarding is enabled:

/var/log/arbitex/siem-dead-letter.jsonl {
    daily
    rotate 30
    compress
    missingok
    notifempty
    copytruncate
}

Audit log extraction for manual SIEM import

In air-gapped environments without a SIEM direct sink configured, audit events accumulate in the local buffer. Extract them manually for import into your SIEM system.

Export the audit log

cp /opt/arbitex-outpost/audit_buffer/audit.jsonl \
   /tmp/arbitex-audit-$(date +%Y%m%d-%H%M%S).jsonl

The file is JSONL format (one JSON object per line). Each entry includes:

Field	Description
`event_id`	Unique event UUID
`outpost_id`	Outpost identifier
`timestamp`	ISO 8601 event time
`request_id`	Request correlation ID
`user_id`	User who made the request
`model`	Target model
`dlp_action`	DLP outcome: `ALLOW`, `BLOCK`, `REDACT`, `PROMPT`
`dlp_entities`	List of detected entity types
`hmac`	HMAC-SHA256 chain signature
`prev_hmac`	Previous entry’s HMAC (chain link)

Verify HMAC chain integrity

Before importing, verify the audit chain has not been tampered with:

# The platform provides an audit verification tool; for manual inspection,
# verify that each entry's hmac field is consistent with its content and prev_hmac.
# See: /admin/audit-log-verification/ for the verification reference.

See Audit log verification for the full verification procedure.

Import into SIEM

Most SIEMs can ingest JSONL directly. For Splunk:

# Using Splunk universal forwarder or one-shot file monitor:
/opt/splunkforwarder/bin/splunk add oneshot /tmp/arbitex-audit-<DATE>.jsonl \
  -sourcetype arbitex:outpost:audit \
  -index arbitex_audit

For Microsoft Sentinel, use the custom log ingestion API or upload the JSONL file via the Log Analytics workspace data upload feature.

Configuration changes

All configuration is managed via /opt/arbitex-outpost/.env. After making changes, restart the outpost:

sudo systemctl restart arbitex-outpost
# or
cd /opt/arbitex-outpost && docker compose -f docker-compose.outpost.yml restart

Common post-install configuration changes

Change	Variable	Notes
Enable Tier 3 DeBERTa DLP	`DLP_DEBERTA_ENABLED=true` + `DEBERTA_MODEL_PATH=<path>`	DeBERTa ONNX file must be accessible inside the container
Enable SIEM direct sink	`SIEM_DIRECT_ENABLED=true` + `SIEM_DIRECT_URL=<url>`	See SIEM direct sink guide
Add GeoIP MMDB	`MAXMIND_DB_PATH=/app/geoip/GeoLite2-City.mmdb`	Copy MMDB to `geoip/` first
Enable HMAC-signed policy bundles	`POLICY_HMAC_KEY=<key>`	Must match the key configured in the cloud portal
Enable GPU inference	Change image tag to `<VERSION>-gpu` in Compose file	Requires NVIDIA runtime