Troubleshooting guide

This guide covers the most common operational issues encountered in Arbitex SaaS portal and Outpost deployments. Each issue follows a Symptom → Diagnosis → Resolution → Prevention pattern where applicable.

Before opening a support ticket, work through the quick diagnostic checklist below. For component-specific deep dives, see the cross-references at the bottom of this guide.

---

Quick diagnostic checklist

Before escalating any issue, answer these five questions. Having clear answers will dramatically reduce resolution time — both for self-service diagnosis and when working with support.

1. What changed recently?

The most common root cause of production issues is a recent change. Check:

• Was a new deployment rolled out? (Outpost binary upgrade, platform release)
• Was a configuration change applied? (hot-reload via SIGHUP, config file edit)
• Was a certificate rotated? (outpost mTLS cert, IdP signing cert, TLS leaf cert)
• Was an IdP reconfigured? (SAML ACS URL, OIDC client secret, SCIM token rotation)
• Was there a network or firewall change?

2. Is the issue reproducible?

• Consistent: every request fails → likely misconfiguration or hard dependency failure.
• Intermittent: some requests succeed → likely a race condition, capacity constraint, or partial upstream failure.
• Time-correlated: started at a specific time → correlate with deployment events or cron jobs.

3. Which environment is affected?

• SaaS portal only (api.arbitex.ai)
• A specific Outpost (identify by outpost_id)
• All Outposts
• Both portal and Outpost

4. What are the exact error messages?

Copy error messages verbatim from:

• Browser developer tools (Network tab → response body)
• UI error banners
• Outpost logs: docker logs arbitex-outpost 2>&1 | tail -200
• Platform audit log: portal → Admin → Audit Events

5. Have you checked the structured logs?

Outpost logs are structured JSON. Use jq to filter relevant events before escalating:

# All ERROR and CRITICAL events in the last 500 lines
docker logs arbitex-outpost 2>&1 | tail -500 | jq 'select(.level == "ERROR" or .level == "CRITICAL")'

# All events from the last 10 minutes
docker logs arbitex-outpost 2>&1 | jq --arg cutoff "$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S)" \
  'select(.timestamp > $cutoff)'

# Specific event type
docker logs arbitex-outpost 2>&1 | jq 'select(.event == "circuit_breaker_state_change")'

See Interpreting structured logs below for a complete jq pattern reference.

---

SaaS portal issues

Login failures

SSO misconfiguration

Symptom: Users are redirected to the IdP login page but never successfully land back in the Arbitex portal. The browser may loop between the IdP and Arbitex, or show a generic error page after the IdP login completes.

Diagnosis:

1. Open browser developer tools (Network tab) and walk through the login. Look for the SAML or OIDC redirect chain and identify where it breaks.

2. For SAML: verify the Assertion Consumer Service (ACS) URL in the IdP configuration. It must be exactly:

   https://api.arbitex.ai/api/auth/saml/acs

   Any difference — trailing slash, http vs https, different subdomain — causes the IdP to post the assertion to a non-existent endpoint or for Arbitex to reject it as not matching the configured ACS URL.

3. For OIDC: verify the redirect URI registered with the identity provider exactly matches the callback URL configured in Arbitex. Mismatches cause the IdP to reject the authorization request.

4. Check for clock skew. SAML assertions include NotBefore and NotOnOrAfter time constraints. The platform allows a ±5 minute window. If the platform server clock is outside this range relative to the IdP, assertions are rejected.

   # Check server clock offset
   chronyc tracking
   timedatectl status

5. Inspect browser cookies. A stale arbitex_session cookie from a previous session can interfere with new login flows. Clear cookies for the Arbitex and IdP domains and retry.

Resolution:

• Correct the ACS URL in the IdP to https://api.arbitex.ai/api/auth/saml/acs.
• For OIDC callback mismatches: update the redirect URI in the IdP application registration to match the Arbitex-configured callback URL exactly.
• Synchronize clocks using NTP. Ensure server time offset stays within ±30 seconds of UTC.
• Clear stale session cookies in affected browsers.

Prevention:

• After every IdP configuration change, verify the ACS URL or redirect URI in both the IdP metadata and the Arbitex SSO settings before testing with real users.
• Monitor SSO authentication success and failure rates via the audit log.
• Set up certificate expiry alerting for the IdP signing certificate (alert 30 days before expiry).

Caution

SAML ACS URL mismatches are the most common cause of SSO redirect loops. Even a trailing slash difference will cause failures for all users until corrected.

Session expiry and token refresh

Symptom: Users are logged out unexpectedly during active sessions. API clients receive 401 Unauthorized responses despite having recently authenticated.

Diagnosis:

1. Check whether the arbitex_session cookie is present and not expired in the browser (Application tab → Cookies in developer tools).

2. For API clients, decode the JWT access token and inspect the exp claim:

   # Decode JWT payload (no signature verification — inspection only)
   echo "<token_payload_segment>" | base64 -d 2>/dev/null | python3 -m json.tool

3. Verify that the client is implementing token refresh before expiry. The access token lifetime is configurable; check the issued expires_in value.

Resolution:

• Browser sessions: re-authenticate. If sessions expire too quickly for your use case, review the session lifetime configuration in the Arbitex admin settings.
• API clients: implement proactive token refresh. Refresh when less than 20% of the access token’s lifetime remains, or when a 401 response is received (attempt one refresh then retry).

Prevention:

• Implement token refresh in all API clients rather than relying on catching 401 errors in production traffic.
• Set session lifetime to match your security policy — shorter is more secure but increases re-authentication frequency.

JWKS rotation: kid mismatch

Symptom: A spike in 401 Unauthorized errors affecting many users simultaneously, with error bodies referencing unknown_key_id or invalid_token. This occurs shortly after a JWKS key rotation event.

Diagnosis:

1. Check the JWKS endpoint for currently published key IDs:

   curl -s https://api.arbitex.ai/api/auth/jwks | python3 -c \
     "import sys,json; [print(k['kid']) for k in json.load(sys.stdin)['keys']]"

2. Decode the JWT header from a failing token and compare its kid claim against the published key IDs:

   echo "<header_segment>" | base64 -d 2>/dev/null

3. Check audit events for a concentration of auth failures at a specific timestamp (the time of JWKS key rotation).

Resolution:

• The platform retains the previous signing key in the JWKS endpoint during a grace period after rotation. If the grace period has elapsed, tokens signed with the old key cannot be validated — affected users must re-authenticate.
• Force clients with cached JWKS to refresh by clearing their local JWKS cache and re-fetching from /api/auth/jwks.
• If the JWKS cache TTL is misconfigured on the client side (too long), reduce it to no more than 1 hour.

Prevention:

• Set JWKS cache TTL to 5–60 minutes. This balances cache efficiency with recovery speed after rotation.
• Alert on 401 error rate spikes. A sudden spike across many users is a strong indicator of JWKS rotation issues rather than individual token problems.

---

DLP rules not applying

Symptom: Content that should be blocked or redacted by a DLP rule passes through without any policy action. The rule appears enabled in the portal.

Diagnosis:

1. Verify the rule’s enabled flag. A rule with enabled: false is visible in the portal but does not evaluate.

2. Check rule priority ordering. Rules are evaluated in ascending priority order (lower number evaluated first) with the first_applicable combining algorithm — the first matching rule wins and subsequent rules are not evaluated. If a higher-priority rule with a broader condition matches first and returns allow, a lower-priority blocking rule will never be reached.

3. Verify entity type targeting. Each DLP rule specifies which entity types it targets. If the content contains an entity type not covered by the rule’s entity type list, the rule will not match.

4. Check the confidence threshold. DLP detections below the configured confidence threshold are not reported as matches. If the content is marginal for detection, it may be detected internally but fall below the threshold configured on the rule.

5. Review the policy action. If the rule is set to log rather than block or redact, the content passes through while the event is recorded. Check the audit log to confirm whether the event appears there.

Resolution:

• Enable the rule if it is disabled.
• Adjust priority ordering so that the blocking rule is evaluated before any conflicting allow rule.
• Add the missing entity type to the rule’s entity type list.
• Lower the confidence threshold on the rule to capture lower-confidence detections.
• Change the rule action from log to block or redact as appropriate.

Prevention:

• After configuring DLP rules, test them with known-positive content (content that should trigger the rule) and verify the expected action is taken.
• Use the audit log to confirm rule evaluation — policy_applied events show which rule matched and which action was taken.

Note

The first_applicable combining algorithm means rule priority order is critical. A broad allow rule at priority 1 will suppress all blocking rules at priorities 2 and above for any request that matches it.

---

Webhook delivery failures

Symptom: Webhook deliveries are failing. The portal shows delivery errors, or events are not arriving at the configured webhook endpoint.

Diagnosis:

1. Check whether the webhook endpoint URL is reachable from the Arbitex platform. The platform’s outbound requests originate from the SaaS infrastructure — ensure your endpoint is publicly reachable or allows traffic from Arbitex IP ranges.

2. Verify TLS certificate validity on the webhook endpoint. The platform requires a valid, trusted TLS certificate on the receiving endpoint. Self-signed certificates are not accepted.

3. Check HMAC payload signature verification. If your endpoint validates the X-Arbitex-Signature header, verify that the shared secret configured in the portal matches the one used in your endpoint’s verification logic exactly.

4. Inspect dead letter files for accumulated failed deliveries:

   tail -20 /var/log/arbitex/{connector}_dead_letter.jsonl | python3 -m json.tool

5. Review HTTP response codes from your endpoint. 4xx responses indicate your endpoint is rejecting deliveries; 5xx or timeouts indicate your endpoint is unavailable or unhealthy.

Resolution:

• Make the webhook endpoint publicly reachable (or update firewall rules to allow Arbitex platform IP ranges).
• Install a valid TLS certificate from a trusted CA on the webhook endpoint.
• Correct the HMAC shared secret — update it in both the portal and your endpoint’s verification code so they match exactly.
• Fix endpoint errors causing 5xx responses.
• After resolving the root cause, trigger a replay of dead letter events.

Prevention:

• Monitor webhook delivery success rates. Alert on elevated delivery failure rates.
• Monitor the dead letter file size. Alert when it exceeds a threshold (e.g., 10 MB or 1,000 entries).
• Test webhook delivery after any endpoint certificate renewal or firewall change.

---

Export rate limit

Symptom: A data export request fails with an error message indicating a rate limit has been reached. Subsequent export attempts also fail.

Diagnosis:

The export feature enforces a 60-second cooldown between export requests per organization. This is a platform-side limit that applies regardless of user account or role.

Resolution:

• Wait 60 seconds after the most recent export attempt before trying again.
• While waiting, prepare the next export configuration (filters, date range, format) so it can be submitted immediately when the cooldown expires.
• For programmatic or high-frequency access to audit data, use the audit events API (GET /api/v1/audit/events) instead of the portal export feature. The API supports pagination and does not enforce the 60-second export cooldown.

Prevention:

• For integrations that require regular data extraction, build against the audit events API rather than the portal export feature from the start.

---

API key authentication failing

Symptom: API requests using an API key return 401 Unauthorized or 403 Forbidden.

Diagnosis:

1. Check the API key status in the portal (Admin → API Keys). A revoked key returns 401. A suspended key also returns 401.

2. Verify the key belongs to the correct organization. API keys are org-scoped — a key issued for org A cannot authenticate requests for org B. Check that the org_id in the request matches the org the key was issued for.

3. Check whether the key has an expiry date configured and whether it has passed. Expired keys return 401.

4. Verify the key is being sent in the correct header format: Authorization: Bearer <api_key>.

Resolution:

• If the key is revoked: generate a new API key in the portal and update all clients using the revoked key.
• If the org_id is mismatched: use the correct key for the target org, or request a key from the correct org.
• If the key is expired: generate a new key. Consider setting a longer expiry or no expiry for long-lived integrations.

Prevention:

• Implement API key rotation as part of your security runbook. Rotate keys before expiry to avoid service interruptions.
• Monitor for 401 response rates on API key authenticated endpoints. A spike may indicate a key was revoked unintentionally.

---

Outpost operator issues

Startup failures

Config validation errors

Symptom: The Outpost container fails to start. Logs show a configuration validation error immediately on startup.

Diagnosis:

1. Run the config validation command to see detailed validation errors before attempting to start the service:

   arbitex-outpost validate-config

2. Check for missing required fields. The three most commonly omitted fields in initial deployments are:

   Field	Description
   outpost_id	UUID identifying this Outpost in the platform
   org_id	Organization UUID this Outpost belongs to
   platform_management_url	URL for the platform management API (e.g., https://api.arbitex.ai)

3. Verify the config file path is correctly mounted into the container. If the container cannot read the config file, it reports a generic startup error rather than a validation error.

Resolution:

• Add all missing required fields to the configuration file.
• Re-run arbitex-outpost validate-config after each correction until it passes cleanly.
• Restart the container once validation passes.

Prevention:

• Run arbitex-outpost validate-config as a step in your deployment pipeline before replacing a running Outpost container.
• Keep a reference config file for each deployed Outpost under version control.

Certificate path errors

Symptom: The Outpost fails to start with an error referencing a certificate file not found or a permission denied error.

Diagnosis:

1. Check the configured certificate paths in the Outpost config:

   Config key	Purpose
   outpost_cert_path	Outpost client certificate (PEM)
   outpost_key_path	Outpost private key (PEM)
   outpost_ca_path	CA certificate bundle for verifying the platform

2. Verify each file exists at the configured path inside the container:

   docker run --rm --entrypoint sh arbitex-outpost:latest \
     -c "ls -la /path/to/cert /path/to/key /path/to/ca"

3. Check file permissions. The process running inside the container must have read access to the certificate and key files. Keys typically require mode 0600 or 0400.

Resolution:

• Correct file paths in the config to match the actual mounted paths inside the container.
• Fix volume mount configuration in docker-compose.outpost.yml so certificate files are mounted at the expected paths.
• Correct file permissions: chmod 0600 /path/to/key.pem.

Prevention:

• Include certificate file path checks in your deployment validation step alongside arbitex-outpost validate-config.
• Use consistent, documented volume mount paths across all Outpost deployments.

DeBERTa model not found

Symptom: The Outpost starts but DLP Tier 3 (contextual DeBERTa validation) is unavailable. Logs show model file not found or model loading errors.

Diagnosis:

1. Check the configured model path:

   arbitex-outpost validate-config | grep -i model

2. Verify the model files exist at the configured path inside the container. DeBERTa requires multiple files (model weights, config, tokenizer).

3. If the model has not been downloaded yet, run the model download command:

   arbitex-outpost download-models

Resolution:

• Download the model using arbitex-outpost download-models and ensure the output directory matches the path configured in outpost_deberta_model_path.
• If the model download must happen during container initialization, use the provided init container or startup script that runs download-models before the main process.
• For environments where internet access is restricted, pre-stage the model files in a private artifact registry and mount them into the container.

Prevention:

• Include model file presence validation in your Outpost deployment checklist.
• For production deployments, build a custom container image with the model files baked in to eliminate runtime download dependencies.

Port conflicts

Symptom: The Outpost fails to start with a “port already in use” or “address already in use” error.

Diagnosis:

1. The admin port defaults to 8300. Check if another process is using this port:

   ss -tlnp | grep 8300

2. If another process occupies the port, identify it and determine whether it should be moved or the Outpost admin port should be reconfigured.

Resolution:

• Stop the conflicting process, or change the Outpost admin_port config value to a free port.
• Note: admin_port is a RESTART_REQUIRED_KEY — a config file change is not sufficient. The container must be restarted with the updated config.

---

DLP detection problems

Entity not detected

Symptom: Content containing sensitive information (e.g., a credit card number, email address, or API key) passes through without any DLP detection or policy action.

Diagnosis:

1. Verify the relevant DLP tier is enabled. Detection is gated by per-tier flags that can be checked via the admin config endpoint (GET http://localhost:8300/admin/config/export):

   Config key	Controls
   dlp_enabled	Master DLP switch — disables all scanning if false
   tier0_enabled	Tier 0: TF-IDF prompt injection pre-filter
   dlp_ner_enabled	Tier 2: GLiNER NER model
   dlp_deberta_enabled	Tier 3: DeBERTa contextual validation
   credint_enabled	Tier 4: Credential Intelligence breach checks

2. Confirm the entity type you expect to be detected is included in the active rule’s entity type list. A rule that targets CREDIT_CARD will not match content containing only EMAIL_ADDRESS.

3. Check the confidence threshold. Detections below the configured threshold are not treated as matches. Lower the threshold temporarily to confirm whether detection is occurring but falling below the bar.

4. Review the combining algorithm. With first_applicable, a prior allow rule at a higher priority level may be consuming the request before the DLP rule is evaluated.

Resolution:

• Enable the required tier flags via hot-reload (SIGHUP) if they are disabled.
• Add the missing entity type to the rule’s entity type list.
• Adjust the confidence threshold.
• Correct rule priority ordering so DLP rules are not preempted by broad allow rules.

Note

dlp_enabled, dlp_ner_enabled, and dlp_deberta_enabled are all RELOADABLE_KEYS. They take effect immediately on SIGHUP without a container restart.

Confidence threshold tuning: too many false positives

Symptom: The DLP pipeline is generating a high volume of false positives — legitimate content is being blocked or flagged incorrectly.

Diagnosis:

1. Review recent DLP audit events. Filter for the entity type generating false positives and inspect the matched text and confidence scores.

2. Identify whether false positives are concentrated in a specific entity type, a specific use case (e.g., technical documentation, code), or a specific user group.

3. For NER-based false positives (Tier 2), check whether GLiNER is misclassifying common words as sensitive entities in context.

4. For regex-based false positives (Tier 1), review the custom pattern against legitimate content samples to confirm over-matching.

Resolution:

• Raise the confidence threshold for the entity type generating false positives. Requires evaluating the tradeoff between false positive rate and detection coverage.
• Add allowlist patterns to exempt known-safe content from matching (e.g., example values in documentation, internal identifiers that match the format of sensitive data).
• For regex false positives: narrow the pattern with word boundaries (\b), anchors, or additional required context characters.

Prevention:

• Test DLP configuration changes against a representative corpus of both positive (should detect) and negative (should not detect) examples before deploying to production.
• Review false positive rates weekly in the DLP analytics dashboard and adjust thresholds proactively.

DeBERTa not loading

Symptom: Tier 3 (DeBERTa contextual validation) is unavailable. Logs show errors during model initialization referencing CUDA, ONNX, or inference device configuration.

Diagnosis:

1. Check the INFERENCE_DEVICE environment variable. Valid values are cuda, cpu, and onnx-cpu.

2. If INFERENCE_DEVICE=cuda, verify CUDA is available in the container:

   docker exec arbitex-outpost python3 -c "import torch; print(torch.cuda.is_available())"

3. If CUDA is unavailable but INFERENCE_DEVICE=cuda, the model will fail to load. Switch to INFERENCE_DEVICE=cpu or INFERENCE_DEVICE=onnx-cpu.

4. Verify the model files at the configured outpost_deberta_model_path are complete and not corrupted:

   docker exec arbitex-outpost ls -la /path/to/model/

Resolution:

• For environments without GPU: set INFERENCE_DEVICE=onnx-cpu. This uses the ONNX Runtime for CPU-only inference — lower throughput than GPU but requires no CUDA dependencies.
• For GPU environments: ensure the CUDA toolkit version in the container matches the installed driver version on the host.
• If model files are corrupted: re-run arbitex-outpost download-models to fetch fresh model files.

Prevention:

• Use INFERENCE_DEVICE=onnx-cpu for all deployments without dedicated GPU resources. GPU acceleration is beneficial for high-throughput deployments but is not required.
• Document the INFERENCE_DEVICE setting in your Outpost deployment runbook alongside GPU/CUDA requirements.

---

Connectivity issues

LLM provider 502/503

Symptom: Outpost requests to the upstream LLM provider (e.g., OpenAI, Azure OpenAI, Anthropic) return 502 Bad Gateway or 503 Service Unavailable.

Diagnosis:

1. Check the upstream provider’s status page. Provider outages are the most common cause of 502/503 errors on the upstream path.

2. Verify the Outpost can reach the provider endpoint from its network location:

   docker exec arbitex-outpost curl -sv https://api.openai.com/v1/models \
     -H "Authorization: Bearer $OPENAI_API_KEY" 2>&1 | head -40

3. Check whether the provider API key is valid and has not expired or been revoked.

4. Review Outpost logs for the specific error details:

   docker logs arbitex-outpost 2>&1 | jq 'select(.level == "ERROR") | select(.upstream != null)'

Resolution:

• If the provider is experiencing an outage: wait for the provider to recover. Consider implementing fallback routing to an alternative provider if uptime requirements demand it.
• If the API key is invalid: rotate the key in the provider portal and update the Outpost configuration.
• If network access is blocked: update firewall egress rules to allow the Outpost to reach the provider endpoint on port 443.

Circuit breaker OPEN

Symptom: Requests through the Outpost fail immediately with a CircuitOpenError or a 503 response referencing a circuit breaker. The circuit breaker was triggered by sustained upstream failures.

Diagnosis:

The Outpost implements a circuit breaker with three states:

State	Meaning	Behavior
CLOSED	Normal operation	Requests forwarded to upstream
OPEN	Upstream failure detected	Requests fail immediately without forwarding
HALF_OPEN	Recovery test in progress	Limited requests forwarded to probe upstream health

1. Check current circuit breaker state via the Prometheus metric:

   outpost_circuit_breaker_state
   # 0 = CLOSED (normal), 1 = OPEN (failing fast), 2 = HALF_OPEN (recovering)

2. Check how many times the circuit has tripped:

   outpost_circuit_breaker_trips_total

3. Review structured logs for state change events:

   docker logs arbitex-outpost 2>&1 | jq 'select(.event == "circuit_breaker_state_change")'

   The event includes breaker, old_state, new_state, and the triggering error.

4. Identify the root cause upstream failure that triggered the open state. The circuit opens after a configured failure threshold is exceeded.

Resolution:

• Fix the upstream issue (provider outage, network connectivity, invalid API key) that caused the circuit to open.
• The circuit breaker will automatically transition to HALF_OPEN after recovery_timeout_seconds and close again if probe requests succeed.
• To force immediate recovery after fixing the upstream issue, restart the Outpost container. This resets the circuit breaker to CLOSED.

Prevention:

• Alert on outpost_circuit_breaker_state > 0 to detect open circuit breakers immediately.
• Review recovery_timeout_seconds in the Outpost configuration to ensure the recovery probe interval is appropriate for your upstream provider’s SLA.

Danger

A circuit breaker in OPEN state means all requests through the Outpost are failing immediately without reaching the LLM provider. This is intentional behavior to protect the upstream from being overwhelmed during an outage, but it means users will experience service interruption until the upstream recovers.

mTLS handshake failure

Symptom: The Outpost cannot establish a management channel connection to the Arbitex platform. Logs show TLS handshake errors. The outpost_tls_handshake_failures_total metric is incrementing.

Diagnosis:

1. Check the TLS handshake failure counter:

   outpost_tls_handshake_failures_total

2. Review Outpost logs for handshake error details:

   docker logs arbitex-outpost 2>&1 | jq 'select(.event == "tls_handshake_failure")'

3. Verify the certificate chain is complete and valid:

   openssl verify -CAfile /path/to/outpost_ca_path /path/to/outpost_cert_path
   openssl x509 -enddate -noout -in /path/to/outpost_cert_path

4. Confirm the CA bundle in outpost_ca_path contains the full chain required to verify the platform’s server certificate.

Resolution:

• Renew expired certificates and update outpost_cert_path and outpost_key_path. Note: these are RESTART_REQUIRED_KEYS — a container restart is required after updating.
• Obtain the correct CA bundle from the platform (Admin → Outposts → Download CA Bundle) and update outpost_ca_path.
• Ensure the outpost client certificate was issued by a CA that the platform trusts for client authentication.

Prevention:

• Monitor outpost_certificate_expiry_seconds and alert when expiry is less than 7 days (604800 seconds).
• Automate certificate renewal as part of your Outpost operations runbook.

Rate limit 429

Symptom: The Outpost is returning 429 Too Many Requests responses to clients. Logs show rate_limit_triggered events.

Diagnosis:

1. Check rate limit configuration:

   Config key	Description
   rate_limit_requests_per_minute	Sustained request rate cap
   rate_limit_burst	Allowed burst above the sustained rate

2. Review the rate_limit_triggered log events to confirm which limit is being hit:

   docker logs arbitex-outpost 2>&1 | jq 'select(.event == "rate_limit_triggered")'

3. Determine whether the rate limit is appropriate for your expected traffic volume, or whether the traffic is genuinely over-limit.

Resolution:

• If the rate limit is too low for legitimate traffic: increase rate_limit_requests_per_minute and rate_limit_burst in the config, then send SIGHUP to the Outpost process. Both keys are RELOADABLE_KEYS — no restart required.

  # Send SIGHUP to reload config (no restart)
  docker kill --signal=SIGHUP arbitex-outpost

• If the traffic is genuinely exceeding safe limits: investigate the source of excess requests (runaway client, misconfigured polling loop) and address it at the client.

Prevention:

• Alert on rate(outpost_requests_total{status_code="429"}[5m]) > 10 to detect sustained rate limit triggering.
• Configure rate limits based on observed peak traffic plus 20-30% headroom.

---

Config issues

Hot-reload not applying

Symptom: A config file change was made and SIGHUP was sent to the Outpost, but the change does not appear to have taken effect. The outpost_config_reloads_by_result_total metric shows a failure, or the changed value is not reflected in GET /admin/config/export.

Diagnosis:

1. Check config reload result metrics:

   outpost_config_reloads_by_result_total{result="failure"}
   outpost_config_reloads_by_result_total{result="success"}

2. Review structured logs for the reload event:

   docker logs arbitex-outpost 2>&1 | jq 'select(.event == "config_reloaded")'

   The event includes changed_keys (keys that changed) and restart_required_keys (keys that changed but require restart — not applied).

3. Verify that the key you changed is a RELOADABLE_KEY. Only these keys can be updated via SIGHUP:

   RELOADABLE_KEYS (take effect immediately on SIGHUP):

   • log_level
   • dlp_enabled
   • dlp_ner_enabled
   • dlp_deberta_enabled
   • credint_enabled
   • budget_enforcement_enabled
   • debug
   • rate_limit_requests_per_minute
   • rate_limit_burst
   • cloud_heartbeat_interval

Resolution:

• If the changed key is a RELOADABLE_KEY and reload is still failing: check for syntax errors in the config file. A malformed config file will cause the reload to fail and the running config to remain unchanged.
• If the changed key is a RESTART_REQUIRED_KEY (see list below): a container restart is required. SIGHUP will not apply the change.

RESTART_REQUIRED_KEYS (require container restart):

• outpost_id
• org_id
• platform_management_url
• outpost_cert_path
• outpost_key_path
• outpost_ca_path
• admin_port
• outpost_api_key

Prevention:

• Before applying a config change, check the key classification above to determine whether a restart is required.
• Validate the config file with arbitex-outpost validate-config before sending SIGHUP.

Tip

To confirm exactly what changed and whether a restart is required, monitor the config_reloaded log event. The restart_required_keys field lists any keys that changed but will only take effect after a restart, serving as a reminder that a restart is still needed.

---

Performance issues

High latency: GPU vs CPU inference

Symptom: DLP scanning adds significant latency to requests. P95 latency is high compared to baseline.

Diagnosis:

1. Check the outpost_dlp_stage_duration_seconds histogram metric broken down by tier to identify which tier is adding latency:

   histogram_quantile(0.95, rate(outpost_dlp_stage_duration_seconds_bucket[5m])) by (tier)

2. If Tier 3 (DeBERTa) latency is high and INFERENCE_DEVICE=cpu, consider whether GPU acceleration is available and feasible.

3. If Tier 1 (regex) latency is unexpectedly high, check for catastrophic backtracking in custom regex patterns.

4. Check system resource utilization:

   docker stats arbitex-outpost --no-stream

Resolution:

• If CPU inference is the bottleneck and GPU is available: set INFERENCE_DEVICE=cuda and restart.
• If GPU is not available: INFERENCE_DEVICE=onnx-cpu provides faster CPU inference than PyTorch CPU mode via the ONNX Runtime.
• For regex latency: audit custom patterns for backtracking and simplify or rewrite problematic patterns.

Prevention:

• For high-throughput deployments (>100 requests/minute), plan for GPU-backed inference or use onnx-cpu to minimize per-request latency.
• Set alerts on outpost_dlp_stage_duration_seconds p95 per tier. Establishing tier-specific baselines makes anomalies easier to detect.

Throughput plateau

Symptom: Outpost throughput stops scaling with added load. Requests queue rather than being processed, and latency increases.

Diagnosis:

1. Check worker thread and concurrency configuration:

   Config key	Description
   WORKER_THREADS	Number of worker threads for request processing
   MAX_CONCURRENT_REQUESTS	Maximum concurrent in-flight requests

2. Check whether the upstream LLM provider is rate-limiting requests (look for 429 responses on the upstream side).

3. Review system CPU and memory utilization. If the host is CPU-saturated, adding more workers will not help and may worsen latency.

Resolution:

• Increase WORKER_THREADS and MAX_CONCURRENT_REQUESTS if the host has available CPU headroom.
• If the upstream provider is rate-limiting: reduce MAX_CONCURRENT_REQUESTS to stay within provider limits and increase rate_limit_requests_per_minute client-side to match.
• For sustained high-throughput needs: scale horizontally by adding additional Outpost instances behind a load balancer.

---

Operational issues

Disk usage alerts

Symptom: A disk_usage_alert log event fires, or the outpost_disk_usage_bytes gauge shows high utilization.

Diagnosis:

1. Check disk usage metrics:

   # Disk used percentage
   outpost_disk_usage_bytes{type="used"} / outpost_disk_usage_bytes{type="total"}

   The metric includes a path label identifying which filesystem is affected.

2. Review log file sizes. Log files are the most common source of unexpected disk growth on Outpost hosts:

   du -sh /var/log/arbitex/
   du -sh /var/log/arbitex/*.jsonl

3. Check dead letter file sizes. Dead letter files grow when delivery is failing — see Webhook delivery failures.

Resolution:

• Configure log rotation if not already in place. The Outpost supports log rotation via its log rotation configuration.
• Truncate dead letter files after replaying or archiving their contents.
• Alert threshold: >90% usage requires immediate attention. >95% is a critical condition.

Prevention:

• Alert on outpost_disk_usage_bytes{type="used"} / outpost_disk_usage_bytes{type="total"} > 0.9.
• Monitor outpost_log_rotation_events_total{result="failure"} to detect failed log rotation before disk fills.

Slow startup / model loading

Symptom: The Outpost container takes a long time to become healthy after startup. The readiness probe fails for an extended period during initialization.

Diagnosis:

The DeBERTa large model requires significant time to load into memory, particularly on CPU. Expected startup times by inference device:

INFERENCE_DEVICE	Typical model load time
cuda (GPU)	15–30 seconds
onnx-cpu	30–60 seconds
cpu (PyTorch)	60–180 seconds

Resolution:

• If startup time is causing health check failures, increase the initial delay in your health check or readiness probe configuration to accommodate model loading time.
• Use GPU (INFERENCE_DEVICE=cuda) for the fastest startup, or INFERENCE_DEVICE=onnx-cpu for a good balance of startup speed and CPU-only compatibility.
• For environments requiring fast restarts, consider running the Outpost on a persistent VM rather than ephemeral containers to benefit from OS-level page cache warming.

---

Interpreting structured logs

The Outpost emits structured JSON logs to stdout. All log lines include timestamp, level, and event fields. Use jq to filter and analyze logs efficiently.

Key event types and jq patterns

Startup events:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "startup")'

Fields: version, outpost_id, org_id, inference_device, dlp_enabled

Scan completion:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "scan_completed")'

Fields: scan_id, scan_duration_ms, entity_types_detected (array), pipeline_tiers_used (array), action_taken

Entity detection:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "entity_detected")'

Fields: entity_type, confidence, tier, scan_id, offset_start, offset_end

Policy applied:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "policy_applied")'

Fields: action, rule_id, combining_algorithm, scan_id, policy_id

Circuit breaker state change:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "circuit_breaker_state_change")'

Fields: breaker, old_state, new_state, trigger_error

Config reloaded:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "config_reloaded")'

Fields: changed_keys (array), restart_required_keys (array), result (success/failure)

Rate limit triggered:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "rate_limit_triggered")'

Fields: client_ip, requests_per_minute, limit

Disk usage alert:

docker logs arbitex-outpost 2>&1 | jq 'select(.event == "disk_usage_alert")'

Fields: path, used_pct, used_bytes, total_bytes

Correlating scan events with policy outcomes

To build a tab-separated timeline of scan results — useful for auditing a specific time window or investigating a pattern of unexpected actions:

docker logs arbitex-outpost 2>&1 \
  | jq -r 'select(.event == "scan_completed") \
    | [.timestamp, .scan_id, (.entity_types_detected | join(",")), .action_taken] \
    | @tsv'

To find all scans that resulted in a block action:

docker logs arbitex-outpost 2>&1 \
  | jq 'select(.event == "scan_completed") | select(.action_taken == "block")'

To correlate an entity detection with its policy outcome using scan_id:

SCAN_ID="<scan_id_from_audit>"
docker logs arbitex-outpost 2>&1 \
  | jq --arg sid "$SCAN_ID" 'select(.scan_id == $sid)'

Log level guide

Level	Meaning	Example events
DEBUG	Verbose development diagnostics	Per-entity scan details, regex match traces
INFO	Normal operational events	startup, scan_completed, config_reloaded
WARNING	Attention needed, service continues	Disk >80%, cert expiry <30d, rate limit approaching
ERROR	Action required, some requests failing	Scan failures, connection errors, auth failures
CRITICAL	Immediate action required, severe impact	Circuit breaker OPEN, disk >95%, OOM, cert expired

Caution

Do not run log_level=DEBUG in production for extended periods. Debug logging includes content fragments in log lines, which may expose sensitive data in log storage. Use DEBUG for targeted diagnostic sessions only and revert to INFO immediately after.

---

Prometheus alerts reference

Recommended alert conditions

The following alert conditions cover the most operationally significant Outpost failure modes. Add these to your Alertmanager configuration:

Metric expression	Condition	Meaning
outpost_disk_usage_bytes{type="used"} / outpost_disk_usage_bytes{type="total"}	> 0.9	Disk >90% full
rate(outpost_tls_handshake_failures_total[5m])	> 5	TLS handshake failures >5/min
rate(outpost_dlp_scans_total{action="error"}[5m]) / rate(outpost_dlp_scans_total[5m])	> 0.01	Scan error rate >1%
outpost_circuit_breaker_state	> 0	Circuit breaker OPEN or HALF_OPEN
outpost_config_reloads_by_result_total{result="failure"}	increase	Config reload failure
rate(outpost_requests_total{status_code="429"}[5m])	> 10	Rate limit 429 rate >10/min
outpost_certificate_expiry_seconds	< 604800	Cert expires within 7 days
outpost_heartbeat_consecutive_failures	> 3	Heartbeat failing

Alertmanager rule examples

groups:
  - name: arbitex_outpost
    rules:

      - alert: OutpostCircuitBreakerOpen
        expr: outpost_circuit_breaker_state > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Outpost circuit breaker is open (outpost={{ $labels.outpost_id }})"
          description: >
            The circuit breaker for outpost {{ $labels.outpost_id }} is in state
            {{ $value }} (1=OPEN, 2=HALF_OPEN). All requests are failing fast.
            Investigate the upstream provider connectivity immediately.
            Metric: outpost_circuit_breaker_state

      - alert: OutpostCertificateExpiringSoon
        expr: outpost_certificate_expiry_seconds < 604800
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Outpost certificate expires within 7 days (outpost={{ $labels.outpost_id }})"
          description: >
            Certificate for outpost {{ $labels.outpost_id }} expires in
            {{ $value | humanizeDuration }}. Renew and redeploy before expiry
            to avoid mTLS handshake failures.
            Metric: outpost_certificate_expiry_seconds

      - alert: OutpostDiskUsageHigh
        expr: >
          outpost_disk_usage_bytes{type="used"}
          / outpost_disk_usage_bytes{type="total"} > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Outpost disk usage exceeds 90% (path={{ $labels.path }})"
          description: >
            Disk usage on path {{ $labels.path }} of outpost {{ $labels.outpost_id }}
            is at {{ $value | humanizePercentage }}. Review log file and dead letter
            file sizes. At >95%, the outpost may stop writing logs or fail entirely.
            Metric: outpost_disk_usage_bytes

      - alert: OutpostHeartbeatFailing
        expr: outpost_heartbeat_consecutive_failures > 3
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Outpost heartbeat failing (outpost={{ $labels.outpost_id }})"
          description: >
            Outpost {{ $labels.outpost_id }} has failed {{ $value }} consecutive
            heartbeats to the platform management API. The outpost will appear
            offline in the portal. Verify network connectivity and platform
            management URL configuration.
            Metric: outpost_heartbeat_consecutive_failures

On-call dashboard metrics

Configure the following metrics in your monitoring dashboard for on-call visibility:

Metric	Visualization	Why it matters
outpost_requests_total by status_code	Rate graph	Overall request volume and error rate by status
outpost_request_duration_seconds p50/p95/p99	Latency histogram	End-to-end request latency trends
outpost_dlp_scans_total by tier and action	Stacked rate graph	DLP processing volume and action distribution
outpost_circuit_breaker_state	Gauge/status panel	Upstream connectivity state at a glance
outpost_heartbeat_success	Uptime indicator	Platform management channel health
outpost_active_connections	Gauge	Current connection concurrency
outpost_certificate_expiry_seconds	Countdown gauge	Time remaining before certificate expiry

---

Getting help

Support channels

• Documentation: https://docs.arbitex.ai — start here for most configuration and operational questions.
• Support portal: https://support.arbitex.ai — submit tickets, track status, escalate.
• Enterprise support: dedicated Slack channel or support engineer contact per your enterprise agreement.

What to include in a support ticket

Providing complete information upfront reduces back-and-forth and accelerates resolution. Include:

1. Environment: SaaS portal, specific Outpost (include outpost_id), or both. Include the Outpost version (arbitex-outpost --version).
2. Error messages: Copy the exact error message verbatim. Include the full JSON log line for structured log errors.
3. Timeline: When did the issue start? What changed before it started? Is it consistent or intermittent?
4. Reproduction steps: A minimal sequence of actions that reliably triggers the issue.
5. Diagnostics bundle: See below.

Exporting diagnostics

Outpost diagnostics:

# Admin audit statistics (port 8300)
curl -s http://localhost:8300/admin/audit/stats | python3 -m json.tool > audit_stats.json

# Current running config (sensitive values redacted by the endpoint)
curl -s http://localhost:8300/admin/config/export | python3 -m json.tool > running_config.json

# Recent structured logs (last 1000 lines)
docker logs arbitex-outpost 2>&1 | tail -1000 > diagnostics.log

# Recent ERROR and CRITICAL events only
docker logs arbitex-outpost 2>&1 | jq 'select(.level == "ERROR" or .level == "CRITICAL")' > errors.json

# Bundle all of the above
tar czf arbitex-diagnostics-$(date +%Y%m%d-%H%M%S).tar.gz \
  audit_stats.json running_config.json diagnostics.log errors.json

SaaS portal diagnostics:

• Use the portal’s data export feature (Admin → Export) to download audit event history.

• For API-level diagnostics, export recent audit events via the API:

  curl -s "https://api.arbitex.ai/api/v1/audit/events?limit=500" \
    -H "Authorization: Bearer <token>" | python3 -m json.tool > portal_audit_events.json

Note

The /admin/config/export endpoint redacts sensitive values (API keys, private key paths) before returning the config. It is safe to share the output with support.

---

Related documentation

Performance and operations:

• Performance tuning guide
• Outpost operations
• Outpost deployment
• Deployment topologies

Monitoring and observability:

• Monitoring and alerting guide
• OpenTelemetry configuration

DLP and policy:

• DLP pipeline configuration
• Policy configuration

Authentication and identity:

• SSO login flow
• Identity providers

Integrations:

• SIEM integration
• Security operations