Outpost Administration

Budget Enforcement

The Outpost enforces per-org budget caps locally before forwarding requests to AI providers. Caps are configured in the policy bundle and evaluated by BudgetCapEnforcer before each request is proxied. When a cap is exceeded the Outpost can block requests, surface a warning header, or silently log the overage — depending on the configured action mode.

Budget enforcement runs entirely on the Outpost. No real-time call to the Platform is required; enforcement decisions use locally-tracked usage from the SQLite UsageTracker. This design keeps enforcement reliable even when the Outpost has degraded Platform connectivity.

How budget enforcement works

Every inbound request to /v1/chat/completions passes through BudgetCapEnforcer.check() before being forwarded:

The enforcer reads budget_config from the current policy bundle.
It calls UsageTracker.get_current_period_totals() to retrieve the running dollar spend and request count for the current billing period.
It computes dollar_percent and request_percent as a fraction of each configured cap.
If usage is 80–99% of any cap, warning=True is set and the proxy adds X-Budget-Warning: approaching to the response.
If usage reaches 100% of any cap, the action configured in action_on_exceed takes effect.
When action_on_exceed=block, the proxy returns HTTP 429 with X-Budget-Status: exceeded.

Fail-open guarantee

If budget_config is absent from the policy bundle (first-time deployment, bundle not yet synced), the enforcer defaults to log_only and never blocks requests. Any unexpected exception inside the enforcer is caught and the request is allowed — a bug in budget enforcement must never take down the proxy.

Billing periods

The budget_config schema supports two period types:

`period` value	Period key format	Reset behaviour
`monthly` (default)	`YYYY-MM`	Resets at the start of each calendar month
`hourly`	`YYYY-MM-DDTHH`	Resets at the top of each clock hour

The UsageTracker derives the period key from the current UTC time. Counters for previous periods are retained in the SQLite database for audit history; only the current-period totals are compared against caps.

Policy bundle configuration

Budget configuration lives in the budget_config key of the policy bundle. Configure caps in the Admin Portal under Settings → Budget — values propagate to all Outposts on the next policy sync (default 60-second interval).

{
  "budget_config": {
    "monthly_dollar_cap": 500.0,
    "monthly_request_cap": 10000,
    "action_on_exceed": "block",
    "period": "monthly"
  }
}

Field	Type	Default	Description
`monthly_dollar_cap`	float	`0` (disabled)	Maximum estimated spend in USD per billing period. Set to `0` or omit to disable the dollar cap.
`monthly_request_cap`	int	`0` (disabled)	Maximum number of requests per billing period. Set to `0` or omit to disable the request cap.
`action_on_exceed`	string	`"log_only"`	Enforcement mode when a cap is exceeded. See Action modes.
`period`	string	`"monthly"`	Billing period type: `"monthly"` or `"hourly"`.

Action modes

Mode	On cap exceeded	HTTP status	Response header	Log level
`block`	Request denied	429	`X-Budget-Status: exceeded`	WARNING
`warn`	Request allowed	200	`X-Budget-Warning: approaching`	WARNING
`log_only`	Request allowed	200	—	INFO

block is recommended for production when strict spend control is required.

warn allows continued service while surfacing the overage — use when visibility is needed without hard cutoffs.

log_only (default when no budget_config is present) records the overage without affecting end-users.

Warning threshold

Regardless of action_on_exceed, when usage reaches 80% of any cap the Outpost adds X-Budget-Warning: approaching to the proxied response. This signals approaching limits before enforcement triggers.

Environment variables

Variable	Default	Description
`USAGE_DB_PATH`	`usage_data/usage.db`	Path to the SQLite database used by `UsageTracker`. Set to empty string to disable local usage tracking entirely (budget enforcement becomes a no-op).

Dollar and request caps are not configurable via environment variables — they are always set through the policy bundle.

Admin API: budget status

The Outpost exposes a budget status endpoint on the admin API (port 8301, localhost-only):

GET /admin/api/budget/status
Authorization: Bearer <admin-token>

Response fields:

Field	Type	Description
`period`	string	Current billing period key (e.g. `2026-03` or `2026-03-12T14`)
`period_type`	string	`monthly` or `hourly`
`dollar_cap`	float	Configured dollar cap (`0` = unlimited)
`dollar_spent`	float	Estimated USD spend in the current period
`dollar_percent`	float	`dollar_spent / dollar_cap × 100` (0 when cap is 0)
`request_cap`	int	Configured request cap (`0` = unlimited)
`request_count`	int	Requests made in the current period
`request_percent`	float	`request_count / request_cap × 100` (0 when cap is 0)
`status`	string	`ok`, `warning` (80–99%), or `exceeded` (≥100%)

Example response:

{
  "period": "2026-03",
  "period_type": "monthly",
  "dollar_cap": 500.0,
  "dollar_spent": 412.33,
  "dollar_percent": 82.5,
  "request_cap": 10000,
  "request_count": 8621,
  "request_percent": 86.2,
  "status": "warning"
}

BudgetPanel — admin UI

The Outpost admin UI (http://localhost:8301) includes the BudgetPanel under the Monitoring section. It shows:

Dollar spend vs. cap — progress bar with warning (amber at 80%) and exceeded (red at 100%) colouring
Request count vs. cap — same visual treatment
Current action_on_exceed badge
Token counts (input + output) for the current period
Auto-refresh every 60 seconds

The panel fetches both /admin/api/budget/status and /admin/api/usage to combine budget enforcement state with raw usage totals.

Multi-model cost calculation

Estimated cost is computed by the UsageTracker using token counts from AI provider responses and per-model pricing from the policy bundle’s cost_rates map. If the provider does not return token counts (e.g. non-buffered streaming path), the usage for that request is recorded as 0 tokens.

Cost estimates may differ from actual provider billing due to provider-specific rounding, mid-period pricing changes, or requests that fail before token counts are returned. Monitor both Arbitex-reported usage and direct provider billing dashboards in production.

Budget troubleshooting

Requests blocked unexpectedly (HTTP 429)

Confirm the Outpost has synced the latest policy bundle. Check last_sync_at in the admin health panel — a stale bundle may contain an outdated lower cap.
Confirm the billing period. Caps reset at period rollover. If the period just rolled over, usage should be near zero.
Review Outpost logs for Budget BLOCK entries — they log the exact totals and period that triggered enforcement.

Usage counters not resetting at period rollover

The UsageTracker resets counters when the period key changes. If the Outpost was stopped at the rollover boundary, stale totals may persist until the next request. Restart the Outpost or wait for the next request — the reset happens on the first check of the new period.

Action mode not taking effect

Confirm the policy bundle has been synced. Check the policy version in the admin health panel.
Verify the cap values are non-zero. A monthly_dollar_cap: 0 with action_on_exceed: block will never trigger — 0 means no cap.

Credential Intelligence

Credential Intelligence (CredInt) detects known-compromised credentials in AI traffic — passwords, API keys, tokens, and connection strings that have appeared in breach data. On the Arbitex SaaS platform, CredInt is provided centrally. On the Outpost, CredInt runs entirely within the customer VPC using a bundled bloom filter, with no credentials leaving the customer environment for lookup.

For the platform-level CredInt overview (what it detects, audit log fields, policy actions), see Credential Intelligence.

What CredInt detects

CredInt adds a fourth tier to the Outpost DLP pipeline. It answers a specific question the earlier tiers cannot: “Has this credential appeared in known breach data?”

The earlier tiers detect credentials by shape and pattern:

Tier 1 (Regex): Structured patterns — SSN formats, credit card numbers, API key formats, connection strings
Tier 2 (NER/spaCy): Named entity recognition — email addresses, usernames, tokens adjacent to credential-shaped context
Tier 3 (DeBERTa): Contextual classification — credential-sharing intent even when patterns don’t match

CredInt (Tier 4) checks the specific values detected by the earlier tiers against the Arbitex breach corpus. A Tier 1 regex match on a password field tells you a password is present. A CredInt hit on that same value tells you the password has already been compromised and is likely in active use by attackers.

DLP cascade with CredInt

Tier 1  Regex          → structured pattern matching (< 1 ms)
Tier 2  NER/spaCy      → named entity recognition (5–50 ms)
Tier 3  DeBERTa        → contextual classification (50–500 ms, optional)
Tier 4  CredInt bloom  → known-compromised credential matching (< 1 ms)

CredInt is placed last in the cascade because:

Tiers 1–3 block or redact many credentials before CredInt is reached. CredInt adds signal for credentials that escape pattern matching — dictionary passwords, novel formats, or values that DeBERTa classifies as non-credentials.
CredInt provides corpus-membership signal that the earlier tiers cannot. “This credential is in the breach corpus” is distinct from “this text contains a credential-shaped string.”
CredInt does not short-circuit the pipeline. It contributes its entity detections and lets the policy resolver determine the final action — consistent with how the other tiers operate.

What triggers a CredInt check

Before querying the bloom filter, a lightweight token extractor identifies credential-shaped tokens in the prompt:

Email-password pairs ([email protected]:password123)
Colon-separated username:password pairs
High-entropy strings adjacent to @ or :// patterns
API key formats not already handled by Tier 1 regex

This pre-filter limits bloom filter lookups per request to a manageable count (typically 0–20 per prompt). Non-credential text never reaches the bloom filter.

Architecture: hybrid bloom filter

The Outpost uses a hybrid bundled + optional CDN refresh architecture (ADR-0001). This design satisfies two competing requirements: air-gap viability (all lookup data must be available offline) and freshness (newly-breached credentials should be detectable as soon as possible).

Bundled filter (baseline)

At Docker image build time, a compressed bloom filter binary is embedded in the image layer. This filter is always present, regardless of network connectivity.

Corpus size: Industry-leading compromised credential dataset
FPR target: 10% (see Bloom filter trade-offs below)
Compressed size: ~440–470 MB
Memory footprint at runtime: ~470 MB RSS
Lookup latency: < 1 μs per entry (O(k) hash operations, k ≈ 5–7)

The bundled filter is fully self-contained. No network calls are made at startup or during request processing.

CDN refresh (internet-connected mode)

When CREDINT_DOWNLOAD_URL is set and the container has internet access, the Outpost attempts to download a fresher filter from Arbitex CDN at startup. If the download succeeds within the configured timeout (default: 45 seconds), the downloaded filter replaces the bundled one in memory. If the download fails or times out, startup continues using the bundled filter — no error, no startup failure.

The downloaded filter carries a metadata header with a corpus snapshot date. If the downloaded filter’s snapshot date is older than the bundled filter, the bundled filter is used (this guards against CDN misconfigurations serving a downgrade).

Air-gap mode

In air-gap deployments, CREDINT_DOWNLOAD_URL is left unset (the default). The bundled filter is loaded at startup. No network calls are made at any point. Full CredInt capability is available from first boot.

Air-gap mode:                    Internet-connected mode:
┌──────────────┐                 ┌──────────────┐
│ Bundled .bf  │                 │ Bundled .bf  │ ← fallback
│ (in image)   │                 │ (in image)   │
└──────┬───────┘                 └──────┬───────┘
       │ loaded at startup              │ if download fails/times out
       ▼                                ▼
  CredInt active                   CDN refresh attempt (45s timeout)
                                        │ if download succeeds
                                        ▼
                                   Downloaded .bf
                                   (newer snapshot)

CredInt configuration

All CredInt configuration is via environment variables in .env.

Variable	Default	Description
`CREDINT_ENABLED`	`true`	Master switch. Set to `false` to disable CredInt entirely.
`CREDINT_BLOOM_PATH`	`/app/credint.bf`	Path to the bundled filter inside the container. Set at image build time — do not change unless supplying a custom filter.
`CREDINT_DOWNLOAD_URL`	`""` (empty)	CDN URL for startup filter refresh. Empty = air-gap mode.
`CREDINT_DOWNLOAD_TIMEOUT_SECONDS`	`45`	Maximum wait for CDN download at startup.
`CREDINT_FPR_THRESHOLD`	`0.10`	Downloaded filters with FPR higher than this value are rejected.

For air-gap deployments, only CREDINT_ENABLED needs attention. The defaults are correct for air-gap.

Helm chart values (Kubernetes deployments)

For Kubernetes deployments using the Outpost Helm chart, the corresponding values in values.yaml:

credint:
  enabled: true
  downloadUrl: ""          # empty = air-gap mode
  downloadTimeoutSeconds: 45

resources:
  limits:
    memory: 2Gi            # Required — bloom filter adds ~470 MB RSS

The memory limit must account for the bloom filter’s runtime footprint. The minimum recommended container memory limit with CredInt enabled is 2 Gi.

Bloom filter trade-offs

The bundled filter is built at 10% false positive rate (FPR). The implications:

FPR vs size

For the current corpus:

FPR	Compressed size	RAM footprint
1%	~1.0–1.1 GB	~1.1 GB
5%	~580–640 MB	~640 MB
10%	~440–470 MB	~470 MB

The 10% FPR filter keeps the image layer within the 5 Gi container image target. Higher FPR choices are available via the CDN refresh path for internet-connected customers who need lower false positive rates.

What 10% FPR means in practice

A 10% FPR means 1-in-10 credential-shaped tokens that are not in the breach corpus will trigger a false CredInt hit. However:

The regex pre-filter (Tier 1) and NER (Tier 2) eliminate most non-credential text before it reaches CredInt. Only tokens that already passed credential-shape heuristics are checked.
Credential-shaped tokens in normal business prompts are rare. The user-visible false positive rate (false alarms per thousand requests) is very low in practice.
The policy action for a CredInt hit is configurable — REDACT (strip the credential from the prompt) or BLOCK (reject the request). REDACT is more tolerant of false positives.

Customers requiring lower FPR in an internet-connected deployment can set CREDINT_DOWNLOAD_URL to pull a 1% FPR filter at startup. This uses more startup time (download + load) but does not affect the image size.

Staleness in air-gap deployments

The bundled filter is static between image releases. The breach-to-detection window equals the image release cadence:

At a weekly release cadence: credentials breached in the past 7 days are not in the filter.
At a monthly release cadence: up to 30 days of new breach data is missing.

This is the same trade-off air-gap customers accept for GeoIP MMDB data (also bundled at image build time). For regulated customers operating in air-gap mode, this staleness window is an accepted security cost — the alternative (any network call for lookup) is disqualifying.

Internet-connected customers using the CDN refresh path receive the latest filter snapshot available at startup (typically updated weekly or on significant breach events), reducing the staleness window significantly without requiring an image rebuild.

CredInt deployment model comparison

Aspect	Air-gap mode	Internet-connected mode
`CREDINT_DOWNLOAD_URL`	Not set	Set to Arbitex CDN URL
Filter source	Bundled in image	CDN download at startup (bundled as fallback)
Filter freshness	Image release cadence	CDN snapshot cadence (typically weekly)
Startup time impact	None	Up to 45s for download + load
Network dependency	None	CDN reachable at startup (optional — failure is non-fatal)
Credential data egress	None	None — only filter binary is downloaded, not credentials
FPR	10% (bundled)	Lower FPR available via CDN

CredInt audit log fields

When CredInt detects a match, the audit entry includes:

Field	Value
`dlp_entities[].type`	`COMPROMISED_CREDENTIAL`
`dlp_entities[].source`	`credint`
`dlp_entities[].tier`	`4`
`dlp_action`	Determined by policy (`REDACT` or `BLOCK`)

For full audit log field reference, see Credential Intelligence.

Health Monitoring

Arbitex Hybrid Outpost reports health to the management plane via a dual-track heartbeat system. Every deployed outpost sends periodic heartbeats both to the Platform management plane (policy sync channel) and to the Cloud Portal (operational dashboard).

Heartbeat Architecture

Two heartbeat channels

┌─────────────────────────────────────────────────────────┐
│                    Hybrid Outpost                        │
│                                                          │
│  HeartbeatSender                                         │
│  ├── Platform heartbeat  ──────────────────────────────► Platform management plane
│  │   POST /v1/orgs/{org_id}/outposts/{outpost_id}/heartbeat │
│  │   Auth: mTLS (same cert as policy sync)              │
│  │   Interval: 120s (with backoff on failure)           │
│  │                                                       │
│  └── Enhanced heartbeat ───────────────────────────────► Cloud Portal
│      POST {CLOUD_HEARTBEAT_URL}/v1/outpost/heartbeat    │
│      Auth: mTLS preferred; Bearer token fallback         │
│      Interval: CLOUD_HEARTBEAT_INTERVAL (default 60s)   │
└─────────────────────────────────────────────────────────┘

Platform heartbeat carries the operational state used for policy sync decisions: version, uptime, policy sync status, DLP tier 3 activation, and pending audit event count.

Enhanced heartbeat carries the richer telemetry displayed in the Cloud Portal dashboard: DLP tier list, certificate expiry, resource usage (CPU/memory/disk).

The enhanced heartbeat fires after every platform heartbeat attempt, regardless of whether the platform heartbeat succeeded.

Backoff on failure

When platform heartbeats fail (network errors, timeouts, HTTP non-2xx), the outpost applies exponential backoff:

delay = min(120s × 2^(failures−1), 900s) × ±10% jitter

First failure: 120s
Second failure: 240s
…capped at 900 seconds (15 minutes)

On the next successful heartbeat the interval resets to the base 120s. Jitter (±10%) prevents thundering-herd reconnection when many outposts recover simultaneously.

Heartbeat Payload Fields

Platform management plane payload

Sent to POST /v1/orgs/{org_id}/outposts/{outpost_id}/heartbeat:

Field	Type	Description
`version`	string	Outpost software version (e.g. `0.1.0`)
`uptime`	int	Seconds since the outpost process started
`policy_version`	string	Version hash of the currently active policy bundle
`last_sync_at`	ISO-8601 string \| null	Timestamp of the most recent successful policy sync
`dlp_model_version`	string	DeBERTa ONNX model identifier, or `none` if Tier 3 is inactive
`pending_audit_events`	int	Approximate count of unsynced audit events (capped at 100); `-1` = error reading count
`tier3_active`	bool	Whether DeBERTa (Tier 3) contextual DLP is currently loaded and available

Cloud Portal enhanced payload

Sent to POST {CLOUD_HEARTBEAT_URL}/v1/outpost/heartbeat:

Field	Type	Description
`outpost_id`	UUID	Outpost identifier
`version`	string	Outpost software version
`uptime_seconds`	int	Seconds since the outpost process started
`last_policy_sync`	ISO-8601 string \| null	Timestamp of the most recent successful policy sync
`dlp_tiers_active`	string[]	Active DLP tiers: subset of `["regex", "ner", "deberta", "credint"]`
`cert_expiry`	ISO-8601 string \| null	mTLS certificate expiry date; null if unreadable
`resource_usage`	object	CPU/memory/disk percentages: `{cpu_percent, memory_percent, disk_percent}`

Platform acknowledgement

The Platform management plane responds to a successful heartbeat with HTTP 200 and optionally a JSON body containing latest_version. If the outpost is running an outdated version:

{"latest_version": "0.2.0"}

The outpost logs a warning: Outpost version outdated: running=0.1.0 latest=0.2.0 — update recommended. No automatic action is taken; the operator must deploy the update.

Health monitoring configuration

Set these environment variables on the outpost:

Variable	Required	Default	Description
`CLOUD_HEARTBEAT_URL`	No	`""`	Base URL of the Cloud Portal heartbeat receiver. When empty, enhanced heartbeats are silently skipped.
`CLOUD_HEARTBEAT_INTERVAL`	No	`60`	Interval in seconds between enhanced heartbeats.
`OUTPOST_CERT_PATH`	Yes (production)	`certs/outpost.pem`	Path to the outpost mTLS client certificate.
`OUTPOST_KEY_PATH`	Yes (production)	`certs/outpost.key`	Path to the outpost mTLS private key.
`OUTPOST_CA_PATH`	Yes (production)	`certs/ca.pem`	Path to the Platform CA certificate for server verification.
`PLATFORM_MANAGEMENT_URL`	Yes	`""`	Platform management plane base URL. Heartbeats are skipped if empty.
`OUTPOST_ID`	Yes	`""`	Outpost UUID from the Cloud Portal registration.
`ORG_ID`	Yes	`""`	Organisation UUID. Required for the heartbeat URL path.

Note: The platform heartbeat interval is hardcoded at 120 seconds and is not configurable via environment variable. The CLOUD_HEARTBEAT_INTERVAL setting applies only to the enhanced (Cloud Portal) heartbeat channel.

Cloud Portal Dashboard

Outpost list (`/portal/outposts`)

The Outposts page in the Cloud Portal shows all registered outposts for the organisation. Each row shows:

Outpost name and region
Last heartbeat timestamp
Status badge (green / amber / red — see thresholds below)
Software version and whether an update is available
Active DLP tiers
Certificate expiry date (with warning when < 30 days remaining)

Status thresholds

Colour	Condition	Meaning
Green (healthy)	Heartbeat received within the last 5 minutes	Outpost is operating normally
Amber (stale)	Last heartbeat 5–30 minutes ago	Outpost may be experiencing connectivity issues or is under high backoff
Red (offline)	No heartbeat for > 30 minutes, or `deregistered` status	Outpost is unreachable or deregistered

Heartbeat history

Navigate to an individual outpost and click Heartbeat History to view the last 50 heartbeat records (paginated, newest first). The admin API behind this view:

GET /v1/admin/outposts/{outpost_id}/heartbeats?limit=50&offset=0
Authorization: X-API-Key <admin-key>

Each record includes: received_at, status, version, uptime_seconds, policy_version, last_sync_at, dlp_tiers_active, cert_expiry, resource_usage.

All-org outpost list (multi-org admins)

GET /v1/admin/outposts
Authorization: X-API-Key <admin-key>

Returns all outposts across all organisations, ordered by most-recently-seen first.

Health monitoring troubleshooting

Outpost shows amber/red but is running

Cause: Heartbeats are reaching the outpost process but not getting through to the Platform or Cloud Portal.

Checks:

Firewall rules. The outpost must be able to make outbound HTTPS connections to PLATFORM_MANAGEMENT_URL and CLOUD_HEARTBEAT_URL. Verify there is no egress firewall blocking TCP 443.
mTLS certificate validity. The outpost will refuse to send heartbeats if OUTPOST_CERT_PATH, OUTPOST_KEY_PATH, or OUTPOST_CA_PATH are missing. Check outpost logs for mTLS certificates required but missing.
Backoff state. After repeated failures the outpost may be sleeping for up to 15 minutes between attempts. Check logs for Heartbeat backoff — sleeping Xs (failure=N). Wait for the next attempt or restart the outpost to reset the backoff counter.
Proxy/load balancer. If the outpost connects via a forward proxy, confirm the proxy allows connections to both the Platform management plane and the Cloud Portal.

Stale status in portal after outpost restart

The Cloud Portal status is derived from last_heartbeat_at. After restart there is a 120-second window before the first platform heartbeat and up to CLOUD_HEARTBEAT_INTERVAL seconds before the first enhanced heartbeat. The status will update automatically once the first heartbeat is received.

Missed heartbeats after policy sync disruption

The heartbeat sender is independent of the policy sync client. A failed policy sync does not prevent heartbeats from being sent. If heartbeats are missing while policy sync is also failing, the root cause is likely a network connectivity issue or an expired mTLS certificate.

Certificate expiry warning in portal

When cert_expiry in the heartbeat is within 30 days, the portal shows a warning badge on the outpost row. Renew the outpost certificate before it expires:

# Via Cloud admin API
POST /v1/orgs/{org_id}/outposts/{outpost_id}/renew
Authorization: X-API-Key <admin-key>

The renewed certificate bundle (cert + key + CA) must be deployed to the outpost’s OUTPOST_CERT_PATH and OUTPOST_KEY_PATH. The outpost process picks up the new cert on next mTLS client creation (next heartbeat cycle after the files are replaced in-place).

Resource usage fields missing from history

resource_usage is populated by the psutil library. If psutil is not installed in the outpost container image, resource fields are omitted from the enhanced heartbeat payload. This does not affect other heartbeat functionality. Install psutil to enable CPU/memory/disk reporting:

RUN pip install psutil

JWT Validation

By default, the Arbitex outpost authenticates proxy requests using API keys. You can additionally enable OAuth JWT validation so that clients can authenticate with M2M Bearer tokens issued by the Arbitex platform (or any compatible OAuth 2.0 authorization server).

When JWT validation is enabled, the outpost accepts requests with an Authorization: Bearer <jwt> header and validates the token’s signature, expiry, and scope before proxying the request.

Requirements

PyJWT library must be installed in the outpost Python environment. If PyJWT is not available, JWT validation is disabled regardless of configuration. Check with python -c "import jwt; print(jwt.__version__)".
RSA public key or JWKS URL must be configured via environment variables.

JWT configuration

JWT validation is controlled by two environment variables. At least one must be set to enable validation.

Option 1 — RSA public key (PEM)

Set OAUTH_JWT_PUBLIC_KEY to a PEM-encoded RSA public key:

export OAUTH_JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
-----END PUBLIC KEY-----"

Or pass it as a file path (set the variable to the file contents):

export OAUTH_JWT_PUBLIC_KEY=$(cat /etc/arbitex/oauth_public_key.pem)

This is the recommended approach for production deployments. The public key must correspond to the private key used to sign JWTs issued by the Arbitex platform or your authorization server.

Option 2 — JWKS URL

Set OAUTH_JWKS_URL to the JWKS endpoint URL of your authorization server:

export OAUTH_JWKS_URL="https://api.arbitex.ai/.well-known/jwks.json"

JWKS URL is fully supported. Keys are fetched on first use and cached with a TTL (default 300 seconds, configurable via OAUTH_JWKS_CACHE_TTL). On cache expiry or a kid miss, the cache is refreshed automatically. If the JWKS fetch fails and cached keys are available, the cached keys continue to be used (fail-open). A threading lock prevents duplicate concurrent fetches.

Checking whether JWT validation is enabled

The outpost admin status endpoint shows operational state. To confirm the outpost is running and accepting requests, use:

GET /admin/api/status

{
  "outpost_id": "550e8400-e29b-41d4-a716-446655440000",
  "policy_version": "bundle-2026-03-14",
  "uptime_seconds": 3600,
  "active_override_count": 0,
  "emergency_kill": false,
  "last_override_modified": "",
  "routing_override": null,
  "update_available": false,
  "latest_version": "1.2.3"
}

JWT validation configuration is set via environment variables (see JWT configuration above) and does not appear as a separate field in the status response. Verify JWT validation is active by confirming OAUTH_JWT_PUBLIC_KEY or OAUTH_JWKS_URL is set in the outpost environment.

Validation process

When the outpost receives an Authorization: Bearer <token> header and JWT validation is enabled, it performs the following checks in order:

Structural check — token must have exactly three dot-separated segments.
Algorithm check — the JWT alg header must be RS256, RS384, or RS512. Tokens signed with any other algorithm (including HS256) are rejected with 401.
Signature verification — the signature is verified against the configured RSA public key.
Expiry check — the exp claim must not be in the past. A 60-second clock skew tolerance is applied to accommodate minor time differences between the issuer and outpost.
Issued-at check — the iat claim is verified with the same 60-second leeway.
Scope check — the scope claim must contain the required scope (default: api:write). The scope claim may be a space-separated string or a JSON array.

Scope requirements

The required scope is api:write by default. Tokens must contain this scope in the scope claim.

The scope claim may be formatted as a space-separated string or a JSON array:

// Space-separated string (RFC 6749 §3.3)
{"scope": "api:write api:read"}

// JSON array
{"scope": ["api:write", "api:read"]}

Both formats are accepted. A token containing api:write in either format passes the scope check.

Fallback to API key authentication

JWT validation is additive, not exclusive. The authentication chain works as follows:

If the Authorization: Bearer header is present and JWT validation is enabled, the token is validated.
- Valid JWT → request proceeds.
- Invalid JWT → 401 or 403 returned immediately. API key fallback is not attempted.
If JWT validation is not enabled, or if no Authorization header is present, the outpost falls through to API key authentication.

This means that disabling JWT validation (by removing the environment variables) transparently reverts to API-key-only authentication with no other configuration changes required.

JWT error responses

Condition	HTTP status	Description
Token expired	`401`	`exp` claim is in the past (after 60s tolerance)
Signature invalid	`401`	RSA signature verification failed
Malformed JWT	`401`	Token cannot be decoded or has wrong structure
Unsupported algorithm	`401`	`alg` header is not RS256, RS384, or RS512
Missing required scope	`403`	`scope` claim does not contain the required scope
JWKS fetch failed	`401`	JWKS endpoint unreachable, returned non-200, or no cached keys available

Obtaining tokens for outpost authentication

Tokens used for outpost JWT authentication must be issued by a compatible authorization server using an RSA signing key. The Arbitex platform M2M token endpoint (POST /api/oauth/token) issues RS256-signed JWTs by default, which are accepted by the outpost JWT validator.

If the platform is configured with OAUTH_JWT_ALGORITHM=HS256 (legacy dev mode only), those tokens use HMAC signing and will not be accepted by the outpost validator, which requires RSA algorithms (RS256, RS384, or RS512). Do not set OAUTH_JWT_ALGORITHM=HS256 in production deployments that use outpost JWT authentication.

JWT security notes

Never set OAUTH_JWT_PUBLIC_KEY to a private key. The outpost only needs the public key for signature verification. Treat the private key as a secret and store it only in your authorization server.
Algorithm restriction is enforced on the outpost side. Even if a JWT header claims alg: none or alg: HS256, the outpost rejects the token. Algorithm confusion attacks are mitigated by the strict allowlist.
Clock skew tolerance is 60 seconds. If outpost system time drifts more than 60 seconds from the issuer, valid tokens may be rejected. Monitor NTP synchronization on outpost nodes.

PVC Recovery

This section covers recovery from outpost PVC (Persistent Volume Claim) failures. The outpost uses three persistent storage areas:

PVC path	Purpose	Loss impact
`policy_cache/`	Cached policy bundle from management plane	Outpost falls back to default policy until re-sync
`audit_buffer/`	HMAC-chained audit events awaiting sync	Unsynced audit events lost (RPO risk)
`dead_letter/`	Failed SIEM delivery batches as JSONL	SIEM gaps until replayed

1. Policy bundle recovery

1a. Connected mode (management plane reachable)

When the outpost has mTLS connectivity to the platform management plane, the policy cache automatically re-syncs every 60 seconds.

# 1. Verify outpost pod is running
kubectl -n arbitex-outpost get pods

# 2. Check policy sync status
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  cat /data/policy_cache/policy_bundle.json | jq '.metadata.version'

# 3. If PVC was recreated, the sync worker will repopulate automatically
#    Watch logs for confirmation:
kubectl -n arbitex-outpost logs deploy/arbitex-outpost -f | grep "policy_sync"

# Expected log line:
# INFO  policy_sync: Bundle refreshed, version=<N>, rules=<count>

If sync fails (certificate expired, network issue):

# Check mTLS connectivity
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  curl -sf --cert /certs/client.pem --key /certs/client-key.pem \
  https://$PLATFORM_MANAGEMENT_URL/v1/internal/policy-bundle | jq '.metadata'

# Common issues:
# - PLATFORM_MANAGEMENT_URL not set or unreachable
# - Client certificate expired → rotate via cert-manager or manually
# - CA bundle mismatch → verify CLOUD_CA_CERT_PATH

1b. Air-gap mode (no management plane)

In air-gapped deployments, the outpost bootstraps from a bundled default policy.

# 1. The outpost uses scripts/default-policy-bundle.json on first boot
#    when no cached bundle exists and PLATFORM_MANAGEMENT_URL is unset

# 2. To update policy in air-gap mode, copy a new bundle into the PVC:
kubectl -n arbitex-outpost cp \
  new-policy-bundle.json \
  arbitex-outpost-0:/data/policy_cache/policy_bundle.json

# 3. Restart the outpost to pick up the new bundle
kubectl -n arbitex-outpost rollout restart deployment/arbitex-outpost

# 4. Verify
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  cat /data/policy_cache/policy_bundle.json | jq '.metadata.version'

Generating an offline policy bundle:

# On a machine with management plane access:
curl -sf --cert client.pem --key client-key.pem \
  https://platform.arbitex.io/v1/internal/policy-bundle \
  -o policy-bundle-export.json

# Transfer to air-gapped environment via approved media
# Then apply as shown above

2. Audit buffer recovery

The audit buffer stores HMAC-SHA256 chained JSONL records. The background sync worker POSTs batches to the platform endpoint /v1/internal/outpost-audit-sync.

2a. Verify buffer state

# Check buffer size
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  wc -l /data/audit_buffer/events.jsonl

# Check HMAC chain integrity
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  python -c "
from outpost.audit.chain import verify_chain
result = verify_chain('/data/audit_buffer/events.jsonl')
print(f'Valid: {result.valid}, Events: {result.count}')
"

2b. Drain buffer after PVC recovery

If the PVC was lost and recreated, the buffer is empty. Any events generated between the last successful sync and the PVC loss are not recoverable from the outpost. Check the platform for the last synced event to identify the gap.

# On the platform side, find the last outpost sync timestamp
curl -sf https://api.arbitex.io/v1/admin/audit?source=outpost&limit=1&order=desc \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq '.[0].timestamp'

If the outpost generated events before PVC loss that were never synced, those events are lost. Document the gap window in the incident report.

2c. Force buffer drain

To force an immediate sync of buffered events:

# Trigger manual sync
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  python -c "
from outpost.audit.sync import AuditSyncWorker
worker = AuditSyncWorker()
result = worker.flush()
print(f'Synced: {result.synced}, Failed: {result.failed}')
"

2d. Buffer ring rotation

The audit buffer has a maximum of 100,000 entries (configurable via max_audit_buffer_entries). When full, the oldest entries are rotated out. Ensure the sync worker is running to prevent data loss from rotation.

3. Dead-letter replay

Failed SIEM delivery batches are stored as JSONL files in dead_letter/. Each file represents a batch that failed to deliver to the configured SIEM endpoint (Splunk HEC or syslog).

3a. Inspect dead-letter queue

# List dead-letter files
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  ls -la /data/dead_letter/

# Sample output:
# -rw-r--r-- 1 app app 245K Mar 10 14:22 batch-20260310-142200.jsonl
# -rw-r--r-- 1 app app 189K Mar 10 14:35 batch-20260310-143500.jsonl

# Inspect a file
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  head -3 /data/dead_letter/batch-20260310-142200.jsonl | jq .

3b. Replay against Splunk HEC

# Replay a single batch
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  bash -c '
for line in $(cat /data/dead_letter/batch-20260310-142200.jsonl); do
  curl -sf -X POST "$SIEM_DIRECT_SPLUNK_URL" \
    -H "Authorization: Splunk $SIEM_DIRECT_SPLUNK_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$line"
done
'

# Replay all dead-letter files
kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  bash -c '
for f in /data/dead_letter/*.jsonl; do
  echo "Replaying $f ..."
  while IFS= read -r line; do
    curl -sf -X POST "$SIEM_DIRECT_SPLUNK_URL" \
      -H "Authorization: Splunk $SIEM_DIRECT_SPLUNK_TOKEN" \
      -H "Content-Type: application/json" \
      -d "$line"
  done < "$f"
  echo "Done: $f"
  mv "$f" "$f.replayed"
done
'

3c. Replay against syslog

kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  bash -c '
for f in /data/dead_letter/*.jsonl; do
  echo "Replaying $f to syslog ..."
  while IFS= read -r line; do
    logger -n "$SIEM_DIRECT_SYSLOG_HOST" \
      -P "$SIEM_DIRECT_SYSLOG_PORT" \
      --rfc5424 -p local0.info "$line"
  done < "$f"
  mv "$f" "$f.replayed"
done
'

3d. Air-gap SIEM replay

In air-gapped environments, dead-letter files must be exported and replayed from a machine with SIEM access.

# 1. Export dead-letter files from outpost
kubectl -n arbitex-outpost cp \
  arbitex-outpost-0:/data/dead_letter/ \
  /tmp/dead-letter-export/

# 2. Transfer to SIEM-connected machine via approved media

# 3. Replay from the connected machine
for f in /tmp/dead-letter-export/*.jsonl; do
  while IFS= read -r line; do
    curl -sf -X POST "$SPLUNK_HEC_URL" \
      -H "Authorization: Splunk $SPLUNK_TOKEN" \
      -H "Content-Type: application/json" \
      -d "$line"
  done < "$f"
  echo "Replayed: $f"
done

3e. Cleanup

After successful replay, remove processed files:

kubectl -n arbitex-outpost exec deploy/arbitex-outpost -- \
  rm /data/dead_letter/*.replayed

4. Full PVC loss recovery checklist

Use this checklist when a CSI driver failure or storage incident causes complete PVC loss.

Security Hardening

This section covers security hardening measures for production Arbitex Hybrid Outpost deployments — JWT lifecycle management, request replay protection, and audit log integrity verification. These features work together to provide defense-in-depth for outpost administration, reducing the attack surface of the admin API and ensuring the integrity of your audit trail.

JWT Expiry and Rotation

Admin JWT tokens have a configurable expiry controlled by the ADMIN_JWT_EXPIRY_SECONDS environment variable. Shortening the token lifetime limits the window of exposure if a token is captured or leaked.

Environment Variables

Variable	Default	Range	Description
`ADMIN_JWT_EXPIRY_SECONDS`	`3600`	300–86400	Time in seconds before an admin JWT expires. Shorter values improve security but require more frequent re-authentication.

Refreshing a Token

The outpost provides a refresh endpoint that allows you to obtain a new token before the current one expires. The refresh window opens 5 minutes before expiry.

Endpoint: POST /admin/api/auth/refresh

Must be called within the 5-minute window before token expiry.
The current token must be included in the Authorization header.
On success, a new JWT with a fresh expiry is returned and the old token is immediately invalidated.
If called outside the refresh window (more than 5 minutes before expiry, or after the token has already expired), the endpoint returns 401 Unauthorized.

Example request:

POST /admin/api/auth/refresh HTTP/1.1
Authorization: X-API-Key <admin-key>
Content-Type: application/json

Example response:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expires_at": "2026-03-15T16:00:00Z",
  "issued_at": "2026-03-15T15:00:00Z"
}

Token Revocation

Tokens can be revoked immediately rather than waiting for natural expiry. The revocation list is checked on every authenticated request.

Endpoint	Method	Description
`/admin/api/auth/revoke`	`POST`	Revoke the current token immediately.
`/admin/api/auth/revocation-list`	`GET`	List all revoked tokens. Returns an array of `{token_id, revoked_at}` objects.
`/admin/api/auth/revocation-list/clear`	`POST`	Remove expired entries from the revocation list. Only removes entries whose original tokens would have already expired based on their issue time and `ADMIN_JWT_EXPIRY_SECONDS`.

Example revocation list response:

[
  {
    "token_id": "a3f2c1d4-e5b6-7890-abcd-ef1234567890",
    "revoked_at": "2026-03-15T13:45:00Z"
  },
  {
    "token_id": "b4e3d2c1-f6a7-8901-bcde-f01234567891",
    "revoked_at": "2026-03-15T14:20:00Z"
  }
]

Request Replay Protection

Replay protection prevents attacks where an adversary captures a valid authenticated request and retransmits it to produce an unintended side effect. When enabled, the outpost maintains a deduplication cache keyed by a client-supplied request ID and rejects any request whose ID has already been processed within the TTL window.

Replay protection environment variables

Variable	Default	Range	Description
`REPLAY_PROTECTION_ENABLED`	`false`	`true`/`false`	Enable request replay protection. When enabled, all POST, PUT, and DELETE requests must include an `X-Request-ID` header.
`REPLAY_PROTECTION_TTL`	`300`	60–3600	Time in seconds that request IDs are retained in the deduplication cache.
`REPLAY_PROTECTION_CACHE_SIZE`	`10000`	1000–1000000	Maximum number of request IDs stored in the deduplication cache. Oldest entries are evicted when the cache is full.

How replay protection works

The client includes an X-Request-ID: <uuid> header on every POST, PUT, and DELETE request.
The outpost checks whether the ID already exists in its deduplication cache.
If the ID is new, the request is processed normally and the ID is stored in the cache with the configured TTL.
If the ID already exists in the cache, the request is rejected with 409 Conflict.
GET requests are not subject to replay protection — they are idempotent and carry no state-mutating side effects.

Replay protection error responses

Duplicate request (409 Conflict):

{
  "error": "duplicate_request",
  "message": "Request ID has already been processed",
  "request_id": "550e8400-e29b-41d4-a716-446655440000"
}

Missing X-Request-ID header when replay protection is enabled (400 Bad Request):

{
  "error": "missing_request_id",
  "message": "X-Request-ID header is required for POST, PUT, and DELETE requests"
}

Stats Endpoint

Use the stats endpoint to monitor deduplication cache activity and confirm that replay protection is operating as expected.

Endpoint: GET /admin/api/replay-protection/stats

Example request:

GET /admin/api/replay-protection/stats HTTP/1.1
Authorization: X-API-Key <admin-key>

Response fields:

Field	Type	Description
`enabled`	boolean	Whether replay protection is currently active.
`cache_size`	int	Number of request IDs currently in the deduplication cache.
`max_cache_size`	int	Maximum cache capacity (`REPLAY_PROTECTION_CACHE_SIZE`).
`ttl_seconds`	int	Current TTL for cached request IDs.
`duplicates_blocked`	int	Total duplicate requests rejected since process start.
`requests_processed`	int	Total unique requests processed since process start.

Example response:

{
  "enabled": true,
  "cache_size": 847,
  "max_cache_size": 10000,
  "ttl_seconds": 300,
  "duplicates_blocked": 3,
  "requests_processed": 24891
}

Audit Log Integrity Verification

The outpost maintains an HMAC chain across all audit log events — each event’s HMAC is computed over its content combined with the previous event’s HMAC. This forms a tamper-evident chain: any modification, deletion, or insertion of records between two existing events will break the chain at that point, making tampering detectable.

Verification Endpoint

Endpoint: GET /admin/api/audit/verify

Example request:

GET /admin/api/audit/verify HTTP/1.1
Authorization: X-API-Key <admin-key>

Response Schema

Field	Type	Description
`verified`	boolean	Whether the verification process completed successfully.
`total_events`	int	Total number of audit events in the chain.
`chain_valid`	boolean	`true` if the entire HMAC chain is intact with no broken links.
`broken_at_event`	int or null	If `chain_valid` is `false`, the event sequence number where the chain break was detected. `null` if the chain is valid.
`first_event_at`	string	ISO 8601 timestamp of the oldest event in the chain.
`last_event_at`	string	ISO 8601 timestamp of the most recent event.
`verification_duration_ms`	float	Time taken to verify the chain in milliseconds.

Example Responses

Healthy chain:

{
  "verified": true,
  "total_events": 15482,
  "chain_valid": true,
  "broken_at_event": null,
  "first_event_at": "2026-01-15T08:00:00Z",
  "last_event_at": "2026-03-15T14:30:00Z",
  "verification_duration_ms": 245.3
}

Broken chain:

{
  "verified": true,
  "total_events": 15482,
  "chain_valid": false,
  "broken_at_event": 8341,
  "first_event_at": "2026-01-15T08:00:00Z",
  "last_event_at": "2026-03-15T14:30:00Z",
  "verification_duration_ms": 312.7
}

Scheduling Verification

Run the verification endpoint on a regular schedule to detect tampering early. A daily cron job or monitoring check is recommended for most deployments; high-security environments should verify more frequently.

Example cron entry (daily at 02:00):

0 2 * * * curl -sf -H "Authorization: X-API-Key $OUTPOST_ADMIN_KEY" \
  https://outpost.internal/admin/api/audit/verify \
  | jq -e '.chain_valid == true' > /dev/null \
  || echo "ALERT: Audit chain broken — investigate immediately" | mail -s "[SECURITY] Outpost audit chain failure" [email protected]

Alert immediately on any response where chain_valid is false. Do not wait for a human to manually review verification output.

Security Hardening Checklist

Use this checklist when preparing an outpost deployment for production or conducting a security review.

JWT Security

Reduce ADMIN_JWT_EXPIRY_SECONDS to 900–1800 for high-security environments
Implement token refresh in all admin tooling to avoid forced re-authentication
Revoke tokens immediately when admin sessions end (POST /admin/api/auth/revoke)
Clear the revocation list periodically to remove stale entries (POST /admin/api/auth/revocation-list/clear)

Replay Protection

Enable REPLAY_PROTECTION_ENABLED=true in production
Ensure all API clients send X-Request-ID headers on mutating requests (UUIDv4 recommended)
Size REPLAY_PROTECTION_CACHE_SIZE for expected request volume — a safe starting point is 10x your peak requests per TTL window
Monitor duplicate counts via GET /admin/api/replay-protection/stats and alert on unexpected spikes

Audit Integrity

Schedule daily audit chain verification (GET /admin/api/audit/verify)
Configure alerting to fire on chain_valid: false
Restrict database access — only the outpost process should have write permissions to audit tables
Back up audit logs to immutable storage (S3 with Object Lock, Azure Immutable Blob Storage)

Plugins and Webhooks

Disable PLUGINS_ENABLED unless custom plugins are actively in use
Verify HMAC signatures on all incoming webhook payloads before processing (X-Webhook-Signature)
Rotate WEBHOOK_EMIT_SECRET on the same schedule as admin API keys
Audit custom redaction patterns for overly broad regex that could mask security-relevant content

Network and Access

Bind the admin API to localhost or an internal network interface only
Use TLS for all admin API access (see Certificate Management)
Rotate admin API keys on a quarterly schedule
Enable outpost firewall rules to restrict admin API access to known, approved IP ranges

SIEM Direct Integration

The Outpost SIEM direct sink (SiemDirectSink) forwards audit events from the Outpost process directly to a SIEM endpoint without routing events through the Arbitex Platform relay. Use this feature when:

Your security policy prohibits audit data from transiting Arbitex Cloud.
You require the lowest possible latency between event generation and SIEM ingestion.
You are operating in an air-gapped network segment that cannot reach the Platform.

Three targets are supported: Splunk HTTP Event Collector (HEC), Microsoft Sentinel Data Collection Rule (DCR), and Elasticsearch Bulk API.

If your Outpost has Platform connectivity and no data-residency constraint, the Platform SIEM connectors in SIEM integration are the simpler path — they require no Outpost configuration.

How the SIEM direct sink works

Configuration source

The SIEM direct sink is configured exclusively via the policy bundle siem_config block — not environment variables. This ensures the same configuration reaches all Outposts without per-host overrides.

{
  "siem_config": {
    "type": "splunk",
    "endpoint": "https://splunk.corp.example.com:8088",
    "token": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "index": "arbitex_audit",
    "enabled": true
  }
}

Batch buffer

Events are accumulated in a local buffer before being flushed to the SIEM endpoint. A flush is triggered when either condition is met:

The buffer reaches 100 events, or
10 seconds have elapsed since the last flush.

This batching reduces per-event network overhead without significantly delaying delivery.

Retry behaviour

If a flush attempt fails (network error, non-2xx response), the sink retries up to 3 times with exponential backoff:

Attempt	Delay before retry
1st retry	1 second
2nd retry	2 seconds
3rd retry	4 seconds

After 3 failed attempts the batch is discarded and the failure is logged. events_failed in the status response increments for each discarded batch.

Fail-open

If the SIEM endpoint is unavailable, the Outpost does not block requests. Audit events that cannot be delivered are dropped after the retry exhaustion. Local audit chain integrity (AUDIT_HMAC_KEY) is unaffected — the HMAC-chained local log continues regardless of sink state.

Event emission

After each audit event is written to the local HMAC-chained log, logger.py calls SiemDirectSink.buffer_event(). The buffer operation is fail-open — a sink exception does not interrupt the main request path.

Splunk HEC configuration

Events are formatted as Splunk HEC JSON payloads with sourcetype: "arbitex:ocsf" — events are formatted in OCSF (Open Cybersecurity Schema Framework) rather than raw JSON.

Policy bundle schema

{
  "siem_config": {
    "type": "splunk",
    "endpoint": "https://splunk.corp.example.com:8088",
    "token": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "index": "arbitex_audit",
    "enabled": true
  }
}

Field	Required	Description
`type`	Yes	`"splunk"`
`endpoint`	Yes	Splunk HEC base URL. The sink appends `/services/collector/event` automatically.
`token`	Yes	HEC authentication token. Sent as `Authorization: Splunk <token>`.
`index`	No	Target Splunk index. Omit to use the HEC token’s default index.
`enabled`	Yes	Set to `true` to activate the sink.

HEC payload format

Events are batched and delivered as a newline-delimited series of HEC JSON objects in a single POST:

{"event": {"class_uid": 4001, "time": 1741478400123, "org": {"uid": "org_abc123"}, "actor": {"user": {"uid": "usr_xyz789"}}, ...}, "sourcetype": "arbitex:ocsf", "source": "arbitex-outpost"}
{"event": {"class_uid": 4001, ...}, "sourcetype": "arbitex:ocsf", "source": "arbitex-outpost"}

The sourcetype is always arbitex:ocsf. Create this sourcetype in your Splunk instance if it does not already exist.

Splunk prerequisites

Splunk Enterprise 8.x+ or Splunk Cloud.
HTTP Event Collector enabled (disabled by default in fresh installs).
An HEC token scoped to the target index.
Network connectivity from the Outpost host to the Splunk HEC port (default 8088).

Microsoft Sentinel configuration

Events are formatted as a JSON array for the Sentinel Data Collection Rule (DCR) Log Ingestion API.

Policy bundle schema

{
  "siem_config": {
    "type": "sentinel",
    "dce_endpoint": "https://myworkspace-XXXX.eastus-1.ingest.monitor.azure.com",
    "dcr_id": "dcr-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "stream_name": "Custom-arbitex_audit_CL",
    "tenant_id": "00000000-0000-0000-0000-000000000000",
    "client_id": "00000000-0000-0000-0000-000000000000",
    "client_secret": "your-app-secret",
    "enabled": true
  }
}

Field	Required	Description
`type`	Yes	`"sentinel"`
`dce_endpoint`	Yes	Data Collection Endpoint URL (from the DCR overview in Azure Portal).
`dcr_id`	Yes	Immutable DCR resource ID (begins with `dcr-`).
`stream_name`	Yes	Custom log stream name configured in the DCR (e.g. `Custom-arbitex_audit_CL`).
`tenant_id`	Yes	Azure AD tenant ID for the app registration.
`client_id`	Yes	App registration (service principal) client ID.
`client_secret`	Yes	App registration client secret.
`enabled`	Yes	Set to `true` to activate the sink.

Authentication

The sink authenticates to the DCR ingestion API using the OAuth 2.0 client credentials flow (tenant_id, client_id, client_secret). Tokens are fetched from https://login.microsoftonline.com/{tenant_id}/oauth2/token.

The app registration must have the Monitoring Metrics Publisher role on the DCR resource.

DCR setup checklist

Create a Log Analytics workspace (or use an existing one).
Create a custom table (arbitex_audit_CL) in the workspace.
Create a DCR with a stream mapping Custom-arbitex_audit_CL → your custom table.
Create an app registration with the Monitoring Metrics Publisher role on the DCR.
Paste the DCE endpoint URL, DCR immutable ID, and stream name into the siem_config block.

Elasticsearch configuration

Events are delivered via the Elasticsearch Bulk API in NDJSON format.

Policy bundle schema

{
  "siem_config": {
    "type": "elastic",
    "endpoint": "https://my-elastic-cluster.example.com:9200",
    "index": "arbitex-audit",
    "token": "ApiKey base64encodedapikeyhere==",
    "enabled": true
  }
}

Field	Required	Description
`type`	Yes	`"elastic"`
`endpoint`	Yes	Elasticsearch base URL (including port).
`index`	Yes	Target index name.
`token`	Yes	Encoded API key. Set as `Authorization: ApiKey <token>`.
`enabled`	Yes	Set to `true` to activate the sink.

Bulk API payload

Events are delivered as NDJSON Bulk API requests:

{"index": {"_index": "arbitex-audit"}}
{"class_uid": 4001, "time": 1741478400123, "org": {"uid": "org_abc123"}, ...}
{"index": {"_index": "arbitex-audit"}}
{"class_uid": 4001, ...}

A single POST to {endpoint}/_bulk delivers the accumulated batch. The sink checks the errors field in the Bulk API response; any shard-level errors increment events_failed.

Admin API: SIEM status

The Outpost exposes a sink status endpoint on the admin API (port 8301, localhost-only):

GET /admin/api/siem/status
Authorization: Bearer <admin-token>

Response fields:

Field	Type	Description
`sink_type`	string	`splunk`, `sentinel`, `elastic`, or `null` if unconfigured
`enabled`	bool	Whether the sink is active
`events_sent`	int	Cumulative count of successfully delivered events since Outpost start
`events_failed`	int	Cumulative count of events dropped after retry exhaustion
`last_error`	string\|null	Last error message, or `null` if no recent errors
`last_flush_at`	string\|null	ISO 8601 UTC timestamp of the last successful flush

Example response:

{
  "sink_type": "splunk",
  "enabled": true,
  "events_sent": 14821,
  "events_failed": 3,
  "last_error": null,
  "last_flush_at": "2026-03-12T14:31:05Z"
}

SIEM status card — admin UI

The Outpost admin UI (http://localhost:8301) includes a SIEM status card in the main Dashboard panel. It displays:

Sink type and enabled state
Events sent and events failed counters
Last flush timestamp
Last error message (if any)

Parallel operation with Platform audit sync

The direct sink runs as an independent background task. It does not replace the audit sync worker that forwards events to the Arbitex Platform — both pipelines receive the same events and operate concurrently.

When both paths are active:

Each event is delivered once to your SIEM directly and once through the Platform relay (which may forward to Platform-configured SIEM connectors).
In air-gapped mode (PLATFORM_SYNC_ENABLED=false), only the direct sink path is active.

Outpost Administration

Budget Enforcement

How budget enforcement works

Fail-open guarantee

Billing periods

Policy bundle configuration

Action modes

Warning threshold

Environment variables

Admin API: budget status

BudgetPanel — admin UI

Multi-model cost calculation

Budget troubleshooting

Requests blocked unexpectedly (HTTP 429)

Usage counters not resetting at period rollover

Action mode not taking effect

Credential Intelligence

What CredInt detects

DLP cascade with CredInt

What triggers a CredInt check

Architecture: hybrid bloom filter

Bundled filter (baseline)

CDN refresh (internet-connected mode)

Air-gap mode

CredInt configuration

Helm chart values (Kubernetes deployments)

Bloom filter trade-offs

FPR vs size

What 10% FPR means in practice

Staleness in air-gap deployments

CredInt deployment model comparison

CredInt audit log fields

Health Monitoring

Heartbeat Architecture

Two heartbeat channels

Backoff on failure

Heartbeat Payload Fields

Platform management plane payload

Cloud Portal enhanced payload

Platform acknowledgement

Health monitoring configuration

Cloud Portal Dashboard

Outpost list (/portal/outposts)

Status thresholds

Heartbeat history

All-org outpost list (multi-org admins)

Health monitoring troubleshooting

Outpost shows amber/red but is running

Stale status in portal after outpost restart

Missed heartbeats after policy sync disruption

Certificate expiry warning in portal

Resource usage fields missing from history

JWT Validation

Requirements

JWT configuration

Option 1 — RSA public key (PEM)

Option 2 — JWKS URL

Checking whether JWT validation is enabled

Validation process

Scope requirements

Fallback to API key authentication

JWT error responses

Obtaining tokens for outpost authentication

JWT security notes

PVC Recovery

1. Policy bundle recovery

1a. Connected mode (management plane reachable)

1b. Air-gap mode (no management plane)

2. Audit buffer recovery

2a. Verify buffer state

2b. Drain buffer after PVC recovery

2c. Force buffer drain

2d. Buffer ring rotation

3. Dead-letter replay

3a. Inspect dead-letter queue

3b. Replay against Splunk HEC

3c. Replay against syslog

3d. Air-gap SIEM replay

3e. Cleanup

4. Full PVC loss recovery checklist

Outpost list (`/portal/outposts`)