Monitoring and alerting guide

This guide covers operational monitoring and alerting setup for the Arbitex platform. For OTel SDK configuration (TracerProvider, MeterProvider, OTLP export, auto-instrumentation), see the OpenTelemetry configuration guide.

---

Prometheus /metrics endpoint

Both the platform and outpost expose a /metrics endpoint in Prometheus 0.0.4 text format. Both endpoints are unauthenticated and intended for scraping from within the cluster or a trusted network.

Platform (port 8080)

The platform uses a dual-layer metrics architecture:

• MetricsMiddleware — instruments every HTTP request using the arbitex.http meter (OTel instruments). Controlled by the METRICS_ENABLED environment variable (default: true).
• PrometheusMetricReader — exports all product metrics from the arbitex.platform meter. This reader is always active when the package is installed and does not require OTEL_EXPORTER_OTLP_ENDPOINT to be set.

MetricsMiddleware instruments:

Metric	Type	Labels	Notes
http_request_count	Counter	method, path, status	All requests
http_request_latency_seconds	Histogram	method, path	All requests
http_error_count	Counter	method, path	5xx responses only

Path normalization: UUIDs in path segments are replaced with :id. The /health and /metrics paths are excluded from instrumentation.

Platform product metrics (arbitex.platform meter):

Metric	Type	Labels	Histogram Buckets
dlp_scan_latency_seconds	Histogram	tier, action	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5
policy_eval_time_seconds	Histogram	policy_id, result	—
provider_response_time_seconds	Histogram	provider, model_id	—
token_count_total	Counter	provider, model_id, type	—
budget_utilization_ratio	Observable Gauge	per-org callback	—

Outpost (port 8300)

The outpost also uses a dual-layer architecture:

• OutpostMetricsStore — stdlib-based metrics store, always available regardless of OTel package installation. Returns 503 if the metrics store was not initialized (startup failure).
• PrometheusMetricReader — OTel-based metrics from the arbitex.outpost meter, available only when OTel packages are installed.

Outpost stdlib metrics (always available):

Metric	Type	Labels
outpost_requests_total	Counter	status_code
outpost_request_duration_seconds	Histogram	—
outpost_dlp_scans_total	Counter	tier, action
outpost_policy_evaluations_total	Counter	result
outpost_active_connections	Gauge	—
outpost_config_reloads_total	Counter	—
outpost_heartbeat_success	Gauge	—

outpost_request_duration_seconds histogram buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0

outpost_heartbeat_success values: 1.0 = ok.

Outpost OTel metrics (arbitex.outpost meter):

Metric	Type	Labels	Notes
outpost_dlp_scan_latency_seconds	Histogram	tier	—
outpost_policy_sync_duration_seconds	Histogram	success	—
outpost_budget_utilization_ratio	Gauge	—	—
outpost_heartbeat_latency_seconds	Histogram	success	—
outpost_audit_buffer_size	Gauge	—	—
outpost_siem_delivery_latency_seconds	Histogram	sink_type	—
outpost_circuit_breaker_open	Gauge	breaker	1.0 = OPEN
outpost_degradation_mode	Gauge	—	1.0 = degraded
outpost_audit_queue_depth	Gauge	—	—
outpost_requests_total	Counter	status	—
outpost_policy_sync_age_seconds	Gauge	—	—

---

Prometheus scrape configuration

Kubernetes ServiceMonitor

If you are running Prometheus Operator, deploy a ServiceMonitor to scrape the platform:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: arbitex-platform
  namespace: arbitex
  labels:
    app: arbitex-platform
    team: platform
    release: prometheus
spec:
  selector:
    matchLabels:
      app: arbitex-platform
  namespaceSelector:
    matchNames: [arbitex]
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s
      honorLabels: true

Create an equivalent ServiceMonitor for the outpost deployment, pointing to port 8300.

Static scrape config (non-Kubernetes)

For bare-metal or Docker Compose deployments, add the following to prometheus.yml:

scrape_configs:
  - job_name: arbitex-platform
    static_configs:
      - targets: ['platform:8080']
    scrape_interval: 30s
  - job_name: arbitex-outpost
    static_configs:
      - targets: ['outpost:8300']
    scrape_interval: 30s

---

Grafana dashboard import

All dashboards are located in deploy/grafana/ in the platform repository. They require Grafana 10.0+ and a Prometheus datasource configured with the variable name DS_PROMETHEUS.

Available dashboards

Dashboard	UID	Key Panels
System Health	arbitex-system-health	Request rate by method, latency p50/p95/p99, error rate 4xx/5xx, active connections, 5xx error ratio, request rate by endpoint
Provider Performance	arbitex-provider-performance	Per-provider latency p50/p95, error rate, token throughput, request distribution pie chart, provider/model summary table
DLP Analysis	arbitex-dlp-analysis	Scan latency p95/p99, trigger rate, scan throughput, latency distribution, trigger rate by tier (regex/ner/deberta/presidio), entity types table
Usage and Billing	arbitex-usage-billing	Chat request rate, token throughput, rate limit rejections, budget utilization gauges, request volume by org, token counts by org
Security Events	arbitex-security-events	Auth failures, rate limit rejections, mTLS failures, IP allowlist blocks, GeoIP anon IP detections, security event timeline, auth failures by reason
Compliance Status	arbitex-compliance-status	Policy violations, audit chain breaks, policy pack evaluations, violations by framework, DLP entity distribution, violation rate by pack

Template variables

Each dashboard uses template variables to filter data. The relevant variables per dashboard:

Variable	Dashboards
$job	System Health
$provider (multi-select)	Provider Performance
$org_id (multi-select)	Usage and Billing
$framework (multi-select)	Compliance Status

Import methods

Grafana UI: Navigate to Dashboards → Import → Upload JSON. Select the dashboard JSON file from deploy/grafana/.

Grafana HTTP API: Use a Grafana admin token and POST to the import endpoint:

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <grafana-admin-token>" \
  -d @deploy/grafana/arbitex-system-health.json \
  http://grafana:3000/api/dashboards/import

File-based provisioning: Copy dashboard JSON files into the Grafana provisioning directory (typically /etc/grafana/provisioning/dashboards/). Grafana will auto-load them on startup or after a configuration reload.

---

Alert rule configuration

Alert rules are defined in deploy/prometheus/alert-rules.yml. All alerts carry the label team: platform.

Alert definitions

Alert	Expression	For	Severity
DLPScanLatencyHigh	histogram_quantile(0.95, sum(rate(dlp_scan_latency_seconds_bucket[5m])) by (le)) > 0.5	5m	warning
ProviderErrorRateHigh	(sum(rate(error_count{path=~"/api/chat.*"}[5m])) / sum(rate(request_count{path=~"/api/chat.*"}[5m]))) > 0.05	5m	critical
BudgetThreshold80	budget_utilization_ratio > 0.80	0m	warning
BudgetThreshold95	budget_utilization_ratio > 0.95	0m	critical
CertExpiryNear	tls_cert_expiry_seconds < (30 * 24 * 3600)	0m	warning
AuditChainBreak	increase(audit_chain_breaks_total[5m]) > 0	0m	critical
InferenceTimeout	histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{handler=~"/api/chat.*"}[5m])) by (le)) > 30	5m	warning
ErrorRateSpike	(sum(rate(http_request_errors_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.01	5m	critical

Loading alert rules

Place alert-rules.yml in your Prometheus rules directory and reference it in prometheus.yml:

rule_files:
  - /etc/prometheus/rules/alert-rules.yml

Reload Prometheus (SIGHUP or the /-/reload API endpoint) after adding the file. Verify that rules are loaded at Status → Rules in the Prometheus UI.

---

Application-level alerting

The platform includes a built-in alerting system independent of Prometheus. This system evaluates metric thresholds directly against platform data and delivers webhook notifications.

Alert rule model

Each alert rule has the following fields:

Field	Type	Notes
name	string	Human-readable alert name
metric_type	enum	cost, error_rate, latency_p95, audit_anomaly, budget_projection, cost_anomaly
threshold	number	Threshold value for comparison
comparison	enum	gt, lt, gte, lte
window_minutes	integer	Evaluation window (default: 60)
webhook_url	string	Webhook delivery target
tenant_id	string (nullable)	Scope to a specific tenant, or null for global
enabled	boolean	Whether the rule is active
cooldown_minutes	integer	Minimum time between re-fires (default: 60)

Metric evaluation

• cost: Sums UsageRecord.cost over the configured window using a SQL aggregate query.
• latency_p95: Queries Message.latency_ms records over the window and computes the 95th percentile in Python.
• error_rate: Reads the MetricsMiddleware in-memory counters and returns a percentage (0–100).

Webhook payload

When an alert fires, the platform sends a POST to the configured webhook_url with the following JSON body:

{
  "alert_name": "High Error Rate",
  "metric_type": "error_rate",
  "metric_value": 5.2,
  "threshold": 5.0,
  "comparison": "gt",
  "triggered_at": "2026-03-14T..."
}

Cooldown behavior

After an alert fires, it will not re-fire until cooldown_minutes have elapsed. This prevents repeated delivery for sustained threshold breaches.

Latency metrics API

GET /api/metrics/latency (admin-only) returns per-model latency statistics with trend data. Supported time windows: 1h, 24h, 7d, 30d. The response includes p50, p95, p99, and average latency per model.

---

Structured logging

Both the platform and outpost emit structured logs and support both JSON and text output formats.

Platform

Environment variables:

Variable	Default	Description
LOG_FORMAT	text	Output format: json or text
LOG_LEVEL	INFO	Minimum log level
SERVICE_NAME	arbitex-api	Service identifier included in JSON output

JSON log schema:

Each JSON log line includes the following fields:

Field	Type	Notes
timestamp	string	ISO-8601 UTC
level	string	Log level
service	string	From SERVICE_NAME
message	string	Log message
logger	string	Logger name
request_id	string	Per-request correlation ID
user_id	string	Authenticated user, if available
tenant_id	string	Tenant context, if available
trace_id	string	OTel trace ID (hex32)
span_id	string	OTel span ID (hex16)

Additional context-specific keys are appended as needed.

The SanitizingFormatter wraps both JSON and text formatters and scrubs sensitive values including email addresses, JWTs, API keys, and IP addresses before log lines are emitted.

When LOG_FORMAT=json, the uvicorn access and error loggers are also wrapped with the JSON formatter.

Outpost

Environment variables:

Variable	Default	Description
LOG_FORMAT	text	Output format: json or text
LOG_LEVEL	INFO	Minimum log level

JSON log schema:

Field	Notes
timestamp	ISO-8601 with milliseconds, UTC
level	Log level
message	Log message
module	Python module name
function	Function name
line	Line number
trace_id	OTel trace ID (hex32), when available
span_id	OTel span ID (hex16), when available
request_id	Per-request correlation ID
outpost_id	Outpost instance identifier
org_id	Organization context

---

Log-trace correlation

Platform

The platform implements log-trace correlation via _ContextVarBridgeProcessor. On each span start, this processor extracts the active trace_id (hex32) and span_id (hex16) from the OTel context and writes them into Python ContextVars. The JsonFormatter reads these ContextVars and includes trace_id and span_id in every log line emitted during the span.

This mechanism works even without an OTLP endpoint configured — it only requires the OTel SDK to be initialized before the first log call.

Outpost

The outpost uses get_trace_context(), which reads the active OTel span and returns a (trace_id_hex32, span_id_hex16) tuple. The AuditLogger calls this function and injects trace context into every audit record it writes. Audit records are HMAC-chained for integrity and synced to the platform and any configured SIEM sinks.

Correlating logs across services

With LOG_FORMAT=json enabled on both platform and outpost, filter log streams by trace_id to view all events from a single end-to-end request. A request that passes through the outpost proxy and then reaches the platform API will carry the same trace_id across both services, provided W3C TraceContext propagation is enabled (the default).

---

Health endpoints (outpost)

The outpost exposes three health endpoints on port 8300.

GET /health

Performs component-level health checks across 7 subsystems:

Component	Critical
proxy	Yes
dlp	Yes
policy_sync	Yes
heartbeat	No
config	No
audit	No
geoip	No

Status values: ok, degraded, unavailable, error

If any critical component (proxy, dlp, or policy_sync) reports unavailable or error, the overall status is unhealthy and the endpoint returns HTTP 503. Otherwise it returns HTTP 200.

Response schema:

{
  "status": "healthy | unhealthy | degraded",
  "version": "x.y.z",
  "components": {
    "<name>": {
      "status": "ok | degraded | unavailable | error",
      "last_check": "<ISO-8601>",
      "<detail-key>": "<detail-value>"
    }
  },
  "warnings": ["<message>"]
}

GET /live

Always returns HTTP 200. Used by liveness probes.

{ "status": "alive", "version": "x.y.z" }

GET /ready

Checks a subset of components for readiness. Critical components for readiness: policy_bundle, dlp_pipeline, audit_logger, proxy_router. Optional components: heartbeat, policy_sync.

Status values: ready, degraded, not_ready

---

Troubleshooting

Metrics endpoint returns 503 (outpost) The outpost metrics store was not initialized. This indicates an outpost startup failure. Check outpost logs for initialization errors.

OTel metrics missing, stdlib metrics present The opentelemetry-exporter-prometheus package is not installed. Install it to enable the arbitex.outpost or arbitex.platform OTel meter metrics alongside the stdlib metrics.

trace_id and span_id are empty in logs The OTel SDK was not initialized before the first log call. Set OTEL_EXPORTER_OTLP_ENDPOINT or ensure OTel packages are installed. Verify that OTel SDK initialization runs at application startup, before any logging occurs.

Grafana dashboards show “No data” Check that the DS_PROMETHEUS datasource variable is configured and points to a reachable Prometheus instance. Verify that the scrape target is up (Prometheus → Status → Targets). Confirm that the Prometheus scrape interval matches the dashboard’s time range expectations.

Alert rules not firing Verify that alert-rules.yml is loaded: navigate to Prometheus → Status → Rules and confirm the rule group is listed. For alerts with for: 5m, the condition must be sustained for the full duration before the alert fires — check whether the condition is intermittent.

budget_utilization_ratio gauge shows 0 budget_utilization_ratio is an Observable Gauge populated via a per-org callback. Ensure the organization has a budget configured. If no budget is configured for an org, the callback returns 0.

Log-trace correlation not working Verify that LOG_FORMAT=json is set. Confirm that the OTel SDK is initialized before the first log call. If using the outpost, verify that W3C TraceContext propagation headers are being forwarded by the proxy.