Advanced Troubleshooting

This guide covers deep-dive diagnostic techniques for operators who have already worked through the Basic Troubleshooting Guide and need to perform root-cause analysis on complex or persistent issues. Where the basic guide follows a symptom-to-resolution pattern, this guide provides the instrumentation, tools, and interpretation frameworks needed to understand why something is failing — not just how to reset it.

For performance-specific tuning after diagnosis, see Performance Tuning. For infrastructure and metrics setup, see Monitoring and Alerting and OpenTelemetry Configuration.

---

1. Debug Logging

Debug mode expands the structured log output significantly beyond what is emitted at info level. It is designed for targeted diagnostic sessions, not continuous production operation.

Enabling debug mode

debug is a reloadable key — it takes effect on SIGHUP without restarting the Outpost process.

1. Open your outpost.yaml configuration file and add or modify the top-level key:

   debug: true

2. Send SIGHUP to the running Outpost process to apply the change:

   # Docker deployment
   docker kill --signal=SIGHUP arbitex-outpost
   
   # Kubernetes deployment
   kubectl exec -n arbitex deploy/arbitex-outpost -- kill -HUP 1
   
   # Direct process
   kill -HUP $(pgrep -f arbitex-outpost)

3. Confirm the reload succeeded by checking for the config_reloaded event:

   docker logs arbitex-outpost 2>&1 | tail -20 | jq 'select(.event == "config_reloaded")'

   The event payload includes a changed_keys array. Verify debug appears in it.

4. Reproduce the issue, collect the relevant logs, then disable debug mode immediately by setting debug: false and sending another SIGHUP.

Danger

Debug logs contain sensitive content. The request_body_preview and response_body_preview fields include the first 200 characters of the raw request and response body — this may include user prompts, AI model completions, PII, or credentials. The dlp_scan_detail field includes per-entity match fragments. Never direct debug log output to a SIEM, persistent log store, or shared observability platform. Capture to a local file, investigate, then delete the capture. Treat debug log files with the same access controls as production secrets.

Additional log fields in debug mode

Field	Type	Description
request_body_preview	string	First 200 characters of the proxied request body
response_body_preview	string	First 200 characters of the proxied response body
dlp_scan_detail	object	Per-tier timing breakdown with entity-level match fragments
dlp_scan_detail.tier1_duration_ms	float	Regex stage elapsed time
dlp_scan_detail.tier2_duration_ms	float	NER stage elapsed time
dlp_scan_detail.tier3_duration_ms	float	DeBERTa stage elapsed time
dlp_scan_detail.entities	array	Each detected entity: type, offset, confidence, matched fragment
policy_eval_trace	array	Full ordered list of policy rules evaluated, with per-rule result and reason
policy_eval_trace[].rule_id	string	Policy rule UUID
policy_eval_trace[].matched	bool	Whether the rule condition was satisfied
policy_eval_trace[].action	string	allow, block, redact, flag
policy_eval_trace[].reason	string	Human-readable explanation of why the rule matched or did not
upstream_tls_info	object	TLS version, cipher suite, and certificate subject for the upstream connection
auth_token_claims	object	Decoded JWT claims used for session resolution (debug only; redacted in info mode)

Filtering debug logs with jq

# Show only policy evaluation traces for blocked requests
docker logs arbitex-outpost 2>&1 | jq 'select(.event == "request_blocked") | {timestamp, request_id, policy_eval_trace}'

# Show DLP scan detail with per-tier timing for slow scans (>100ms total)
docker logs arbitex-outpost 2>&1 | jq '
  select(.dlp_scan_detail != null)
  | select(
      (.dlp_scan_detail.tier1_duration_ms + .dlp_scan_detail.tier2_duration_ms + .dlp_scan_detail.tier3_duration_ms) > 100
    )
  | {timestamp, request_id, dlp_scan_detail}'

# Show all entity matches for a specific request ID
docker logs arbitex-outpost 2>&1 | jq --arg rid "req_abc123" '
  select(.request_id == $rid and .dlp_scan_detail != null)
  | .dlp_scan_detail.entities[]'

# Summarize which policy rules are blocking most frequently
docker logs arbitex-outpost 2>&1 | jq -r '
  select(.policy_eval_trace != null)
  | .policy_eval_trace[]
  | select(.matched == true and .action == "block")
  | .rule_id' | sort | uniq -c | sort -rn | head -20

Complete structured log field reference (info + debug)

The following fields appear at info level and above. Fields marked (debug) require debug: true.

Field	Always present	Description
timestamp	yes	RFC3339 UTC timestamp
level	yes	DEBUG, INFO, WARN, ERROR, CRITICAL
event	yes	Machine-readable event name
outpost_id	yes	Outpost UUID
request_id	on proxy events	Unique per-request identifier for correlation
user_id	on proxy events	Authenticated user UUID
provider_id	on proxy events	Target AI provider
dlp_action	on DLP events	allow, block, redact, flag
circuit_breaker_state	on CB events	CLOSED, OPEN, HALF_OPEN
config_reload_id	on reload events	UUID for this reload operation
changed_keys	on reload events	Array of config keys that changed value
request_body_preview	(debug)	First 200 chars of request body
response_body_preview	(debug)	First 200 chars of response body
dlp_scan_detail	(debug)	Per-tier timing and entity fragments
policy_eval_trace	(debug)	Full rule evaluation chain

---

2. SAML/OIDC Trace Analysis

Authentication flow failures are among the most opaque issues in production. This section provides a systematic approach to reading SAML and OIDC flows at the protocol level. See also Identity Providers and SSO Login Flow.

SAML trace with browser developer tools

1. Open the browser developer tools (F12) and navigate to the Network tab. Enable Preserve log so the capture persists across redirects. Clear existing entries.

2. Initiate the SAML login flow from the beginning (navigate to the Arbitex portal login page and click the SSO button).

3. In the Network tab, filter by saml or look for POST requests to the ACS endpoint: https://api.arbitex.ai/api/auth/saml/acs

4. Select the POST to the ACS endpoint. In the Payload tab, locate the SAMLResponse form field. Copy the value.

5. Decode the base64-encoded assertion:

   # Paste the SAMLResponse value into a file first
   echo "PASTE_SAML_RESPONSE_HERE" | base64 -d | xmllint --format - 2>/dev/null

6. In the decoded XML, verify the following critical fields:

   # Extract and check NotBefore / NotOnOrAfter (clock skew tolerance: ±5 minutes)
   echo "$saml_decoded" | grep -E 'NotBefore|NotOnOrAfter'
   
   # Extract NameID and NameIDFormat
   echo "$saml_decoded" | grep -E 'NameID|Format'
   
   # Verify Destination matches the Arbitex ACS URL
   echo "$saml_decoded" | grep 'Destination'

7. Common failure signatures in the decoded assertion:

   Symptom	What to check
   SAML assertion has expired	NotOnOrAfter is in the past; check IdP and Outpost host clock sync
   Assertion not yet valid	NotBefore is in the future; clock skew exceeds ±5 minutes
   Invalid ACS URL	Destination does not match https://api.arbitex.ai/api/auth/saml/acs
   NameID format mismatch	Format in assertion does not match what Arbitex expects (emailAddress or persistent)
   Signature verification failed	IdP signing certificate has rotated; update in portal → Admin → Identity Providers

Verifying SAML signing certificates

# Extract the signing certificate from the SAML response
echo "$saml_decoded" | grep -A 20 'X509Certificate' | \
  grep -v 'X509' | tr -d ' \n' | \
  fold -w 64 | \
  awk 'BEGIN{print "-----BEGIN CERTIFICATE-----"} {print} END{print "-----END CERTIFICATE-----"}' \
  > /tmp/saml_signing.pem

# Inspect certificate validity and subject
openssl x509 -in /tmp/saml_signing.pem -noout -text | grep -E 'Subject:|Not Before|Not After'

# Compare fingerprint against what is configured in the portal
openssl x509 -in /tmp/saml_signing.pem -noout -fingerprint -sha256

Caution

IdP certificate rotation is a common source of SAML breakage. Many IdPs rotate signing certificates on a schedule (quarterly or annually) without notification. Set a calendar reminder to verify certificate expiry dates at least 30 days in advance. After rotation, update the certificate in portal → Admin → Identity Providers and re-download the Arbitex SP metadata to provide the updated ACS URL and SP certificate to the IdP.

OIDC authorization code flow trace

1. Open browser developer tools → Network tab. Enable Preserve log.

2. Initiate the OIDC login flow. Watch for the following request sequence:

   • GET /api/auth/oidc/authorize — Arbitex redirects the browser to the IdP authorization endpoint
   • GET <IdP>/authorize?... — IdP prompts for login
   • GET /api/auth/oidc/callback?code=...&state=... — IdP redirects back to Arbitex with authorization code

3. Inspect the callback request. The code parameter should be present. A missing code with an error parameter indicates the IdP rejected the authorization request. Common error values:

   error value	Meaning
   access_denied	User denied consent, or IdP group/role restriction
   invalid_client	Client ID mismatch; verify in portal → Admin → Identity Providers
   invalid_scope	Requested scopes not configured on the IdP application
   server_error	IdP-side failure; check IdP logs

4. If the callback succeeds but the user is not logged in, the token exchange failed. Check Outpost logs for oidc_token_exchange_error:

   docker logs arbitex-outpost 2>&1 | jq 'select(.event == "oidc_token_exchange_error")'

5. Common token exchange failures:

   Log field	Failure mode
   error: "invalid_grant"	Authorization code expired or already consumed; check latency between callback and exchange
   error: "invalid_client"	Client secret mismatch; rotate and update in portal
   tls_error	Outpost cannot reach the IdP token endpoint; check network connectivity and certificate trust

IdP-specific considerations

Azure AD / Entra ID: Metadata refresh interval defaults to 24 hours. After changing application configuration (redirect URIs, claims, certificates), force a metadata refresh or wait up to 24 hours for propagation.

Okta: SAML certificate rotation requires manually downloading and re-uploading the new certificate to Arbitex. Okta does not expose a metadata URL that Arbitex can poll automatically in all configurations.

Google Workspace: SAML assertions from Google always use emailAddress as the NameID format. Ensure the Arbitex IdP configuration matches. Clock skew issues are rare but confirm NTP is active on the Outpost host.

---

3. Network Capture and Analysis

Network-level analysis is essential for diagnosing TLS failures, DNS resolution problems, and intermittent connectivity issues that do not surface clearly in application logs. See also Outpost Operations and Distributed Tracing.

Capturing outpost-to-provider traffic

Outpost-to-provider traffic is TLS-encrypted. tcpdump captures are useful for timing analysis, connection establishment patterns, and TCP-level failure detection — not for reading payload content.

# Capture traffic from the Outpost container to external providers
# Replace eth0 with the actual interface and adjust the port as needed
tcpdump -i eth0 -w /tmp/outpost-capture.pcap \
  'host api.openai.com or host api.anthropic.com' &

# Run the capture for 60 seconds while reproducing the issue
sleep 60 && kill %1

# Analyze the capture — look for TCP RST (connection reset) or SYN without SYN-ACK (blocked)
tcpdump -r /tmp/outpost-capture.pcap -nn 'tcp[tcpflags] & (tcp-rst) != 0'

# Check for connection timeouts (SYN with no response)
tcpdump -r /tmp/outpost-capture.pcap -nn 'tcp[tcpflags] == tcp-syn'

mTLS handshake debugging

Arbitex Outpost uses mutual TLS for cloud-to-outpost management traffic. Certificate or chain issues surface as handshake failures.

1. Identify the management endpoint and port. The outpost admin listens on port 8300 by default.

2. Test the TLS handshake with openssl s_client, providing the client certificate and key:

   openssl s_client \
     -connect <outpost-host>:8300 \
     -cert /path/to/outpost.crt \
     -key /path/to/outpost.key \
     -CAfile /path/to/ca.crt \
     -servername <outpost-host> \
     -verify_return_error \
     2>&1 | head -60

3. Interpret the output:

   • Verify return code: 0 (ok) — handshake succeeded; TLS is not the problem
   • certificate verify failed — certificate chain issue; check CA bundle and certificate expiry
   • no shared cipher — TLS version or cipher suite mismatch
   • Connection refused — process not listening; check Outpost is running and admin port is correct
   • Connection timed out — firewall blocking the port

4. Verify the certificate chain is complete:

   openssl verify -CAfile /path/to/ca.crt /path/to/outpost.crt

5. Check certificate expiry on all three files (cert, key pair subject, CA):

   for cert in outpost.crt ca.crt; do
     echo "=== $cert ==="; openssl x509 -in $cert -noout -enddate
   done

DNS resolution inside containers

DNS failures inside containers are a frequent source of connectivity issues that look identical to firewall blocks in application logs.

Docker

# Run DNS diagnostic from inside the Outpost container
docker exec arbitex-outpost nslookup api.openai.com
docker exec arbitex-outpost dig api.anthropic.com +short

# Check which DNS server the container is using
docker exec arbitex-outpost cat /etc/resolv.conf

# Test resolution latency
docker exec arbitex-outpost dig api.openai.com | grep 'Query time'

Kubernetes

# Run DNS diagnostic from inside the Outpost pod
kubectl exec -n arbitex deploy/arbitex-outpost -- nslookup api.openai.com
kubectl exec -n arbitex deploy/arbitex-outpost -- dig api.anthropic.com +short

# Verify CoreDNS is healthy
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Proxy and firewall interference detection

Corporate proxies and next-generation firewalls can silently intercept, modify, or drop TLS connections.

# Check if TLS interception is occurring by comparing certificate fingerprints
# The certificate presented to the Outpost should be the actual provider certificate
openssl s_client -connect api.openai.com:443 -servername api.openai.com 2>/dev/null | \
  openssl x509 -noout -issuer -subject

# If the issuer is your corporate CA (not DigiCert, Let's Encrypt, etc.),
# TLS inspection is active — the CA bundle on the Outpost host must include your corporate CA

# Test with explicit SNI to detect SNI-based routing rules
openssl s_client -connect api.openai.com:443 -servername api.openai.com -status 2>&1 | \
  grep -E 'Verify|issuer|subject'

Distributed tracing and correlation IDs

Every Outpost proxy request generates a request_id that is propagated through the OpenTelemetry span tree. See Distributed Tracing for full setup instructions.

# Find the request_id for a failing request
docker logs arbitex-outpost 2>&1 | jq --arg uid "user_abc123" \
  'select(.user_id == $uid and .event == "request_blocked") | {timestamp, request_id, dlp_action}'

# Pull all spans for a given trace ID from your tracing backend (Jaeger example)
# Replace TRACE_ID with the value from the request_id field
curl "http://jaeger:16686/api/traces/TRACE_ID" | jq '.data[0].spans[] | {operationName, duration}'

---

4. DLP Performance Profiling

Use these techniques when DLP latency is elevated but the cause is not obvious from high-level metrics. See also DLP Pipeline Configuration and Performance Tuning.

Per-tier latency analysis

The outpost_dlp_stage_duration_seconds histogram exposes per-tier timing bucketed by stage label. Query this metric to isolate which tier is contributing to latency.

# 95th percentile latency by DLP tier (last 5 minutes)
histogram_quantile(0.95,
  sum by (le, stage) (
    rate(outpost_dlp_stage_duration_seconds_bucket[5m])
  )
)

Expected P95 ranges by stage and device:

Stage	CUDA	ONNX-CPU	PyTorch CPU
tier1_regex	~1 ms	~1 ms	~1 ms
tier2_ner	~10 ms	~15 ms	~20 ms
tier3_deberta	5–15 ms	30–80 ms	100–300 ms

If Tier 1 P95 exceeds 5 ms, investigate regex pattern complexity (see below). If Tier 3 is within expected range but overall latency is high, check chunking behavior using outpost_dlp_scan_tokens_total.

Diagnosing Tier 1 regex catastrophic backtracking

Catastrophic backtracking occurs when a regex engine explores an exponentially large number of paths before failing a non-match. It manifests as periodic CPU spikes correlated with specific request content.

Symptoms:

• tier1_duration_ms spikes to hundreds of milliseconds on specific requests
• Spike correlates with long strings matching a partial prefix of a pattern
• CPU usage spikes on the Outpost container for seconds at a time

Diagnosis:

# Identify which requests triggered slow Tier 1 scans using debug mode
docker logs arbitex-outpost 2>&1 | jq '
  select(.dlp_scan_detail.tier1_duration_ms > 10)
  | {timestamp, request_id, tier1_ms: .dlp_scan_detail.tier1_duration_ms,
     preview: .request_body_preview}'

Common backtracking patterns to avoid:

Problematic pattern	Why	Safer alternative
(a+)+b	Nested quantifiers on overlapping groups	a+b
(\w+\s*)+end	Repeated groups that can match empty	\w+(\s+\w+)*\s*end
.*foo.*bar.*	Unbounded leading .*	Anchor: ^.*?foo.*?bar or split into ordered checks
`(a	aa)+b`	Alternation with overlapping options

Fix: Review custom DLP regex patterns in portal → Admin → DLP Rules. Add anchors (^, $) where possible, avoid nested quantifiers, and test patterns against adversarial inputs using an online regex debugger with backtracking visualization before deploying.

Benchmarking DeBERTa inference

# Check which device DeBERTa is using
docker logs arbitex-outpost 2>&1 | jq 'select(.event == "deberta_device_selected")'

# Query average inference time per token batch
curl -s http://localhost:8300/metrics | grep outpost_dlp_stage_duration_seconds

# Calculate throughput: scans per second at current token distribution
docker logs arbitex-outpost 2>&1 | jq -r '
  select(.dlp_scan_detail.tier3_duration_ms != null)
  | .dlp_scan_detail.tier3_duration_ms' | \
  awk '{sum+=$1; n++} END {printf "avg_ms=%.1f scans_per_sec=%.1f\n", sum/n, 1000/(sum/n)}'

DeBERTa model memory footprint by device:

Configuration	Memory
Float32 (PyTorch CPU or CUDA)	~400 MB
ONNX quantized (CPU)	~200 MB

Tip

If running on CPU-only hosts, prefer dlp_deberta_device: onnx-cpu over pytorch-cpu. ONNX Runtime with quantization delivers 30–80 ms/scan vs. 100–300 ms/scan for PyTorch CPU, with half the memory footprint. See DLP Pipeline Configuration for device configuration options.

Correlating token count with latency

# Scatter: token count vs scan duration (use Grafana scatter plot panel)
rate(outpost_dlp_scan_tokens_total[5m])

# Identify requests that triggered chunking (>512 tokens)
# These will show 2x or 3x Tier 3 duration relative to single-chunk requests
histogram_quantile(0.99, rate(outpost_dlp_stage_duration_seconds_bucket{stage="tier3_deberta"}[5m]))

If P99 is significantly higher than P95, large requests are likely triggering multi-chunk inference. Consider adding a max_token_limit policy to cap request size upstream, or increasing Outpost concurrency to parallelize chunk processing.

---

5. Circuit Breaker Tuning

The Outpost circuit breaker protects downstream AI providers from thundering-herd retries when a provider is degraded or unavailable. See also Outpost Reliability and Outpost Operations.

State machine deep dive

The circuit breaker for each provider operates as a three-state machine:

CLOSED ──[failure_threshold exceeded]──► OPEN
  ▲                                        │
  │                                  [recovery_timeout]
  │                                        │
  └──[half_open_max_requests succeed]── HALF_OPEN

• CLOSED (metric value: 0): Normal operation. Requests flow through. Failures are counted.
• OPEN (metric value: 1): Provider tripped. All requests immediately return 503. No upstream traffic.
• HALF_OPEN (metric value: 2): Recovery probe. A limited number of requests (half_open_max_requests) are allowed through to test whether the provider has recovered. If they succeed, transitions to CLOSED. If they fail, resets to OPEN with a fresh recovery_timeout.

Configuration parameters

circuit_breaker:
  failure_threshold: 5          # consecutive failures before OPEN
  recovery_timeout_seconds: 30  # seconds in OPEN before attempting HALF_OPEN
  half_open_max_requests: 3     # probe requests allowed in HALF_OPEN

Monitoring circuit breaker state

# Current state by provider (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
outpost_circuit_breaker_state

# Trip rate — how often the breaker is opening
rate(outpost_circuit_breaker_trips_total[1h])

# Time spent in OPEN state per provider (approximation via state changes)
changes(outpost_circuit_breaker_state[1h])

# Real-time: watch circuit breaker state changes in logs
docker logs -f arbitex-outpost 2>&1 | jq 'select(.event == "circuit_breaker_state_change")'

Tuning for provider failure patterns

Flaky provider (intermittent)

Flaky providers fail occasionally but recover quickly. Overly aggressive breaker settings cause unnecessary 503s.

circuit_breaker:
  failure_threshold: 10         # tolerate more failures before tripping
  recovery_timeout_seconds: 15  # try recovery sooner
  half_open_max_requests: 5     # more probe requests before deciding recovered

Use this profile when P95 provider latency is normal but error rate fluctuates between 0–5%.

Hard-down provider

Providers that go completely unavailable should be tripped quickly and given adequate recovery time.

circuit_breaker:
  failure_threshold: 3          # trip fast on hard failures
  recovery_timeout_seconds: 60  # longer recovery window
  half_open_max_requests: 2     # minimal probing — failures are likely to continue

Use this profile for providers with known maintenance windows or those behind load balancers that may return 503 en masse.

Force-reset procedures

If a circuit breaker is stuck in OPEN state after a provider has recovered (for example, after a maintenance window), force a reset via the admin API:

# List current circuit breaker states
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8300/admin/circuit-breakers | jq '.'

# Force reset a specific provider's circuit breaker to CLOSED
curl -s -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"provider_id": "openai", "state": "CLOSED"}' \
  http://localhost:8300/admin/circuit-breakers/reset

Caution

Force-resetting a circuit breaker bypasses the recovery probe mechanism. Only do this when you have confirmed through independent means (provider status page, direct connectivity test) that the provider is healthy. If the provider is still degraded, the breaker will re-trip immediately under load.

---

6. Dead Letter Queue Management

The dead letter file captures failed asynchronous event deliveries: webhook delivery failures that have exhausted all retries, and audit event forwarding failures when the SIEM or log pipeline is unavailable. See also Webhook Operations and Monitoring and Alerting.

Inspecting dead letter contents

# Default dead letter file location
DEAD_LETTER_FILE="/var/lib/arbitex-outpost/dead_letter.jsonl"

# Count entries by type
jq -r '.type' "$DEAD_LETTER_FILE" | sort | uniq -c | sort -rn

# Show the 10 most recent entries
tail -10 "$DEAD_LETTER_FILE" | jq '.'

# Show only webhook delivery failures
jq 'select(.type == "webhook_delivery")' "$DEAD_LETTER_FILE"

# Show only audit forwarding failures
jq 'select(.type == "audit_event_forward")' "$DEAD_LETTER_FILE"

# Inspect failure reasons
jq '{type, failed_at, reason, attempt_count}' "$DEAD_LETTER_FILE" | head -40

Monitoring dead letter growth

# Dead letter file size (should not grow continuously)
outpost_dead_letter_file_size_bytes

# Alert if dead letter file exceeds 50MB — indicates persistent delivery failure
outpost_dead_letter_file_size_bytes > 52428800

Replay procedures

1. Before replaying, diagnose and resolve the underlying delivery failure. Replaying into a still-broken endpoint will just re-dead-letter the same events.

2. Verify the webhook endpoint is reachable and returning 2xx:

   curl -s -o /dev/null -w "%{http_code}" -X POST \
     -H "Content-Type: application/json" \
     -d '{"test": true}' \
     https://your-webhook-endpoint.example.com/arbitex

3. Replay failed webhook deliveries via the admin API:

   # Replay all dead-lettered webhook events
   curl -s -X POST \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     http://localhost:8300/admin/webhooks/replay | jq '.'
   
   # The response includes a replay job ID and count of events queued

4. Monitor the replay job progress:

   docker logs -f arbitex-outpost 2>&1 | \
     jq 'select(.event == "webhook_replay_delivered" or .event == "webhook_replay_failed")'

5. After a successful replay, the dead letter file entries are removed automatically. Verify:

   wc -l "$DEAD_LETTER_FILE"

Purging dead letters safely

If dead-lettered events cannot be replayed (endpoint permanently decommissioned, events too stale to be useful), purge them after confirming with your security/compliance team that the events are not needed for audit purposes.

# Back up before purging
cp "$DEAD_LETTER_FILE" "$DEAD_LETTER_FILE.backup.$(date +%Y%m%d%H%M%S)"

# Purge via admin API (preferred — atomic, logged)
curl -s -X DELETE \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8300/admin/dead-letter

# Confirm the file is empty
wc -l "$DEAD_LETTER_FILE"

---

7. Config Change Rollback

Not all configuration errors are obvious at reload time. Some issues (incorrect policy logic, wrong rate limits) only surface under production traffic. This section covers the full safe hot-reload workflow and rollback procedures.

Safe hot-reload workflow

1. Back up the current running configuration:

   cp /etc/arbitex-outpost/outpost.yaml \
     /etc/arbitex-outpost/outpost.yaml.backup.$(date +%Y%m%d%H%M%S)

2. Apply the configuration change to outpost.yaml.

3. Validate the configuration before reloading:

   arbitex-outpost validate-config --config /etc/arbitex-outpost/outpost.yaml

   The validator checks required fields, type correctness, and known constraint violations (for example, admin_port conflicts). Validation does not catch all semantic errors (for example, a typo in a provider URL) but catches syntax and schema issues.

4. Send SIGHUP to apply the change:

   docker kill --signal=SIGHUP arbitex-outpost

5. Confirm the reload succeeded and identify which keys changed:

   docker logs arbitex-outpost 2>&1 | tail -30 | \
     jq 'select(.event == "config_reloaded") | {reload_id, changed_keys, status}'

6. Verify the change took effect by checking the specific behavior (test a request, check a metric, verify a log field).

7. If the change caused a regression, immediately roll back:

   cp /etc/arbitex-outpost/outpost.yaml.backup.<timestamp> \
     /etc/arbitex-outpost/outpost.yaml
   docker kill --signal=SIGHUP arbitex-outpost

Reloadable vs restart-required keys

Understanding which keys take effect on SIGHUP vs requiring a container restart prevents unnecessary downtime.

Reloadable keys (SIGHUP applies immediately):

log_level, dlp_enabled, dlp_ner_enabled, dlp_deberta_enabled, credint_enabled, budget_enforcement_enabled, debug, rate_limit_requests_per_minute, rate_limit_burst, cloud_heartbeat_interval

Restart-required keys (container restart needed):

outpost_id, org_id, platform_management_url, outpost_cert_path, outpost_key_path, outpost_ca_path, admin_port, outpost_api_key

Caution

Changing a restart-required key with SIGHUP will log a config_reload_warning event indicating the key change was detected but cannot be applied without restart. The Outpost will continue running with the previous value for that key. Check the log after every reload to confirm no restart-required changes were silently deferred.

Rolling back restart-required changes

If a container image upgrade or restart-required config change causes a regression:

1. Identify the previous working image tag from your deployment history:

   docker inspect arbitex-outpost --format '{{.Config.Image}}'
   # or check docker-compose.yml git history
   git log --oneline -10 -- docker-compose.yml

2. Pin the previous image tag in docker-compose.yml:

   services:
     arbitex-outpost:
       image: arbitex/outpost:1.2.3  # pin to last known good

3. Restore the previous configuration backup.

4. Restart the container:

   docker compose up -d arbitex-outpost

5. Verify startup succeeds:

   docker logs arbitex-outpost 2>&1 | tail -30 | \
     jq 'select(.event == "outpost_started")'

---

8. SCIM Provisioning Debugging

SCIM sync failures can leave user accounts in inconsistent states. This section covers how to diagnose sync failures at the protocol level. See also SCIM Provisioning.

Diagnosing SCIM sync failures

1. Check the audit log for SCIM events. In the portal navigate to Admin → Audit Events and filter by event type scim_*, or query via API:

   curl -s -H "Authorization: Bearer $API_KEY" \
     "https://api.arbitex.ai/api/v1/admin/audit-events?event_type=scim_sync_error&limit=20" | \
     jq '.events[] | {timestamp, error_code, detail, user_external_id}'

2. Common error codes and their causes:

   Error code	Cause	Resolution
   SCHEMA_MISMATCH	IdP sent an attribute in an unexpected format or namespace	Check SCIM schema mapping in portal → Admin → Identity Providers → SCIM
   MISSING_REQUIRED_ATTRIBUTE	userName or emails not present in provisioning payload	Verify IdP attribute mapping includes userName and primary email
   DUPLICATE_USER	Two IdP users resolved to the same userName	Investigate at IdP; may require deduplication before next sync
   QUOTA_EXCEEDED	User count would exceed plan limit	Upgrade plan or reduce provisioned user count
   INVALID_GROUP_REF	Group membership reference points to a non-existent group	Ensure groups are provisioned before users that reference them

Testing SCIM with curl

Use manual SCIM pushes to isolate whether a failure is in the IdP push logic or the Arbitex SCIM handler.

# Variables
SCIM_BASE="https://api.arbitex.ai/api/scim/v2"
SCIM_TOKEN="<your-scim-bearer-token>"

# Create a test user
curl -s -X POST "$SCIM_BASE/Users" \
  -H "Authorization: Bearer $SCIM_TOKEN" \
  -H "Content-Type: application/scim+json" \
  -d '{
    "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"],
    "userName": "[email protected]",
    "name": {"givenName": "Test", "familyName": "User"},
    "emails": [{"value": "[email protected]", "primary": true}],
    "active": true
  }' | jq '{id, userName, status: .meta.resourceType}'

# Retrieve the user to verify round-trip
curl -s "$SCIM_BASE/Users?filter=userName+eq+%[email protected]%22" \
  -H "Authorization: Bearer $SCIM_TOKEN" | jq '.Resources[0] | {id, userName, active}'

# Patch the user (deactivate)
USER_ID="<id from create response>"
curl -s -X PATCH "$SCIM_BASE/Users/$USER_ID" \
  -H "Authorization: Bearer $SCIM_TOKEN" \
  -H "Content-Type: application/scim+json" \
  -d '{
    "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
    "Operations": [{"op": "replace", "path": "active", "value": false}]
  }' | jq '{id, active}'

Schema mismatch diagnosis

If the IdP is sending attributes that Arbitex does not recognize, enable debug logging (see Section 1) and look for scim_request_body_preview in the debug output to see the exact payload the IdP is sending. Compare against the SCIM 2.0 core schema and the enterprise user extension used by Arbitex.

# In debug mode, capture the raw SCIM request payload
docker logs arbitex-outpost 2>&1 | jq '
  select(.event == "scim_user_create" or .event == "scim_user_update")
  | {timestamp, event, request_body_preview}'

---

Cross-references

Topic	Guide
Basic symptom-to-resolution troubleshooting	Troubleshooting Guide
Frequently asked questions	FAQ
DLP pipeline stages and configuration	DLP Pipeline Configuration
DLP performance tuning levers	Performance Tuning
OpenTelemetry setup and exporters	OpenTelemetry Configuration
Distributed tracing and span analysis	Distributed Tracing
Outpost day-2 operations	Outpost Operations
Identity provider configuration	Identity Providers
End-to-end SSO login flow	SSO Login Flow
SCIM user provisioning	SCIM Provisioning
Webhook configuration and delivery	Webhook Operations
Prometheus alerts and dashboards	Monitoring and Alerting
TLS, mTLS, and certificate management	Security Architecture