Metrics & Monitoring

Metrics & Monitoring

This guide covers the complete observability stack for the Arbitex platform and outpost: Prometheus metrics endpoints, metric definitions, Grafana dashboard setup, alert rule configuration, and OpenTelemetry distributed tracing integration.

Overview

The Arbitex platform exposes Prometheus-compatible metrics from two independent components:

• Platform — the cloud-hosted API gateway, built with FastAPI. Uses the OpenTelemetry SDK with a PrometheusMetricReader bridge to expose metrics at /metrics in Prometheus text exposition format.
• Outpost — the self-hosted proxy agent. Uses a dual-layer approach: a hand-rolled in-memory metrics store for low-overhead counters and gauges, layered with an OTel SDK registry for richer instrumentation. Both the main app and the admin app expose /metrics endpoints.

All /metrics endpoints are auth-bypassed and return the Prometheus text exposition format, compatible with any standard Prometheus scrape job.

Tip

If you run the outpost behind a firewall, ensure your Prometheus scraper can reach both the main outpost port (default 8080) and the admin port (default 9090). They expose overlapping but non-identical metric sets.

---

Platform Metrics

HTTP Request Metrics

The platform instruments every HTTP request through a metrics middleware layer. UUID and ULID path segments are normalized to :id before recording, preventing cardinality explosion from high-volume endpoints. The paths /health and /metrics are excluded from instrumentation.

Metric	Type	Labels	Description
http_request_count	Counter	method, path, status	Total HTTP requests received
http_request_latency_seconds	Histogram	method, path	Request latency from receipt to response completion
http_error_count	Counter	method, path	Total HTTP 5xx server errors

The status label on http_request_count carries the numeric HTTP status code as a string (e.g., "200", "429", "503").

DLP and Product Metrics

These metrics cover the data loss prevention pipeline, policy evaluation engine, LLM provider interactions, and budget enforcement.

Metric	Type	Labels	Buckets	Description
dlp_scan_latency_seconds	Histogram	tier, action	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5	DLP scan latency per tier and action
policy_eval_time_seconds	Histogram	policy_id, result	Default OTel boundaries	Policy evaluation duration
provider_response_time_seconds	Histogram	provider, model_id	Default OTel boundaries	LLM provider response latency
token_count_total	Counter	provider, model_id, type	—	Token usage; type is input or output
budget_utilization_ratio	Observable Gauge	org_id	—	Current budget utilization as a ratio from 0.0 to 1.0

The tier label on DLP metrics corresponds to the scan tier configured in your policy chain (e.g., tier1, tier2). The action label reflects the DLP disposition (allow, redact, block).

Caution

budget_utilization_ratio is an observable (pull-based) gauge. It is computed at scrape time by querying the budget store. Under very high scrape frequency this adds measurable latency. The default scrape interval of 15–60 seconds is appropriate.

Platform Prometheus Endpoint

Method	Path	Description
GET	/metrics	Returns all OTel-bridged metrics via prometheus_client.generate_latest()

---

Outpost Metrics

The outpost exposes a larger set of metrics covering its proxy, DLP pipeline, policy engine, circuit breakers, heartbeat subsystem, air-gap operations, certificate management, and infrastructure. Metrics are grouped below by functional area.

Core Metrics

Metric	Type	Labels	Description
outpost_requests_total	Counter	status_code	Total HTTP requests proxied through the outpost
outpost_request_duration_seconds	Histogram	—	End-to-end request duration at the outpost boundary
outpost_dlp_scans_total	Counter	tier, action	Total DLP scans by tier and resulting action
outpost_policy_evaluations_total	Counter	result	Total policy evaluations by outcome (allow, deny, redact)
outpost_active_connections	Gauge	—	Current number of active inbound connections
outpost_config_reloads_total	Counter	—	Total configuration reload events
outpost_heartbeat_success	Gauge	—	Heartbeat status: 1 = last heartbeat succeeded, 0 = failed

DLP and Detection Metrics

Metric	Type	Labels	Description
outpost_dlp_stage_duration_seconds	Histogram	tier	Duration of each DLP pipeline stage, labeled by tier
outpost_dlp_scan_duration_seconds	Histogram	provider, action	End-to-end DLP scan duration, including provider round-trip
outpost_dlp_entity_detections_total	Counter	entity_type, tier	Count of entity detections by entity type and tier
arbitex_scan_tier_match_count_total	Counter	tier, entity_type	Per-tier match counts; tier is regex, ner, or deberta

Use outpost_dlp_entity_detections_total to track which PII entity types are most frequently encountered. The entity_type label values correspond to the entity taxonomy defined in your DLP policy configuration.

The arbitex_scan_tier_match_count_total metric provides per-tier match breakdowns. Use it alongside outpost_dlp_scans_total to calculate match rates per tier:

sum by (tier) (rate(arbitex_scan_tier_match_count_total[5m]))
/
sum(rate(outpost_dlp_scans_total[5m]))

Policy and Cache Metrics

Metric	Type	Labels	Description
outpost_policy_evaluations_by_action_total	Counter	action, direction	Policy evaluations broken down by action and request direction (request, response)
outpost_policy_evaluations_by_algorithm_total	Counter	action, combining_algorithm	Policy evaluations broken down by action and combining algorithm
outpost_policy_cache_hits_total	Counter	—	Policy evaluation cache hits
outpost_policy_cache_misses_total	Counter	—	Policy evaluation cache misses

Cache hit ratio can be derived as:

rate(outpost_policy_cache_hits_total[5m])
/
(rate(outpost_policy_cache_hits_total[5m]) + rate(outpost_policy_cache_misses_total[5m]))

Circuit Breaker Metrics

Metric	Type	Labels	Description
outpost_circuit_breaker_state	Gauge	breaker	Current state: 0 = closed (healthy), 1 = open (tripped), 2 = half-open (probing)
outpost_circuit_breaker_trips_total	Counter	breaker	Total number of times the circuit tripped to OPEN state
outpost_circuit_breaker_probe_total	Counter	circuit_name, result	Probe attempts made while in half-open state, with result = success or failure

Tip

Alert on outpost_circuit_breaker_state == 1 for any circuit breaker to detect platform connectivity or provider failures quickly. The breaker label identifies the specific subsystem (e.g., platform_api, audit_writer).

Heartbeat and Connectivity Metrics

Metric	Type	Labels	Description
outpost_heartbeat_rtt_seconds	Histogram	—	Round-trip latency of heartbeat calls to the platform
outpost_heartbeat_consecutive_failures	Gauge	—	Current count of consecutive heartbeat failures without a success

Air-Gap Metrics

These metrics are only populated when the outpost is running in air-gap mode with offline policy bundle updates.

Metric	Type	Labels	Description
outpost_airgap_last_sync_timestamp	Gauge	—	Unix epoch timestamp of the last successful bundle sync
outpost_airgap_pending_updates	Gauge	—	Number of policy updates queued and not yet applied
outpost_airgap_bundle_age_seconds	Gauge	—	Age of the currently loaded policy bundle in seconds

Caution

In air-gap deployments, alert on outpost_airgap_bundle_age_seconds > 900 (15 minutes) to detect stale policy bundles before they represent a compliance risk.

Certificate Metrics

Metric	Type	Labels	Description
outpost_certificate_expiry_seconds	Gauge	cert_type	Seconds remaining until the certificate expires; cert_type is mtls_client, mtls_ca, or tls_server

Infrastructure Metrics

Metric	Type	Labels	Description
outpost_tls_handshake_failures_total	Counter	reason	TLS handshake failures, labeled by failure reason
outpost_config_reloads_by_result_total	Counter	result	Config reloads broken down by result (success, failure)
outpost_config_reload_duration_seconds	Histogram	—	Duration of configuration reload operations
outpost_disk_usage_bytes	Gauge	path, type	Disk usage for outpost-managed paths; type is used or free
outpost_log_rotation_events_total	Counter	result	Log rotation events by result (success, failure)
outpost_shutdown_state	Gauge	—	Current shutdown state: 0 = running, 1 = draining, 2 = stopped

Connection Pool Metrics

Metric	Type	Labels	Description
outpost_connection_pool_active	Gauge	—	Number of connections currently in use
outpost_connection_pool_idle	Gauge	—	Number of idle connections available in the pool
outpost_connection_pool_exhaustion_total	Counter	—	Total number of connection pool exhaustion events

Pool exhaustion events indicate that the outpost is receiving more concurrent requests than the pool can serve. Tune the pool size or scale outpost replicas if this counter increases steadily.

Proxy Metrics

Metric	Type	Labels	Description
outpost_request_size_bytes	Histogram	—	Request body size in bytes
outpost_proxy_latency_seconds	Histogram	provider, model	Latency for proxied requests, broken down by provider and model

Outpost Prometheus Endpoints

The outpost exposes two independent /metrics endpoints:

App	Endpoint	Metric Source
Main app	GET /metrics	Hand-rolled in-memory metrics store
Admin app	GET /metrics	OTel SDK registry; point-in-time gauges are refreshed immediately before each scrape

Both endpoints are auth-bypassed and return Prometheus text exposition format. Configure separate Prometheus scrape jobs for each port if you run both.

---

Grafana Dashboard Setup

Data Source Configuration

Add a Prometheus data source in Grafana with the following settings:

Setting	Value
Name	DS_PROMETHEUS
Type	Prometheus
URL	Your Prometheus server URL
Scrape interval	Match your prometheus.yml global scrape interval

Configure Prometheus scrape jobs to pull from all three metric endpoints:

prometheus.yml

scrape_configs:
  - job_name: arbitex-platform
    static_configs:
      - targets: ['<platform_host>:8000']
    metrics_path: /metrics

  - job_name: arbitex-outpost
    static_configs:
      - targets: ['<outpost_host>:8080']
    metrics_path: /metrics

  - job_name: arbitex-outpost-admin
    static_configs:
      - targets: ['<outpost_admin_host>:9090']
    metrics_path: /metrics

Pre-Built Dashboards

Pre-built Grafana dashboard JSON files are available in the platform repository at monitoring/grafana/dashboards/. Each dashboard is parameterized to use the DS_PROMETHEUS data source variable.

Dashboard	File	Key Panels
System Health Overview	health-overview.json	Request rate, error rate, latency percentiles (p50/p95/p99)
DLP Pipeline	dlp-pipeline.json	Per-tier latency (p50/p95/p99), scan volume, action breakdown
Provider Performance	provider-performance.json	Provider latency by model, token throughput
Usage Metering	usage-metering.json	Token usage by org, budget utilization, request count
Security Events	security-events.json	Auth failures, 401/403 rates, error volume over time
Compliance	compliance.json	Policy evaluation time, violation count, DLP action rates

To import a dashboard:

1. Open Grafana and navigate to Dashboards → Import.
2. Click Upload JSON file and select the dashboard file from monitoring/grafana/dashboards/.
3. On the import screen, set the data source to DS_PROMETHEUS.
4. Click Import.

Tip

If your Grafana instance manages multiple environments, add an env label to your Prometheus scrape jobs and use a Grafana variable filter on env within the dashboard to isolate production vs. staging metrics.

---

Alert Rules

Platform Alerts

Alert rules for platform metrics are defined in monitoring/prometheus/alerts/platform-alerts.yml. Load this file via the rule_files key in your prometheus.yml.

Alert Name	Expression	For	Severity	Description
DLPScanLatencyWarning	histogram_quantile(0.95, rate(dlp_scan_latency_seconds_bucket[5m])) > 0.5	5m	warning	p95 DLP scan latency exceeds 500ms
DLPScanLatencyCritical	histogram_quantile(0.95, rate(dlp_scan_latency_seconds_bucket[5m])) > 1.0	5m	critical	p95 DLP scan latency exceeds 1 second
HTTPErrorRateHigh	rate(http_error_count_total[5m]) / rate(http_request_count_total[5m]) * 100 > 5	5m	critical	HTTP 5xx error rate exceeds 5%
RequestLatencyP99High	histogram_quantile(0.99, rate(http_request_latency_seconds_bucket[5m])) > 2	5m	warning	p99 request latency exceeds 2 seconds
BudgetUtilizationCritical	budget_utilization_ratio > 0.9	5m	critical	Budget utilization exceeds 90% for any org
ProviderErrorRateHigh	Error rate > 10% on provider chat endpoints	5m	warning	LLM provider error rate is elevated
ProviderResponseTimeSlow	histogram_quantile(0.99, rate(provider_response_time_seconds_bucket[5m])) > 5	5m	warning	p99 provider response time exceeds 5 seconds

Recommended Outpost Alerts

These thresholds are derived from the outpost admin metrics-summary response and represent recommended operational baselines. Adjust them to match your deployment’s SLOs.

Signal	Yellow Threshold	Red Threshold
DLP scan latency (p95)	> 100ms	> 500ms
Policy sync duration	> 1s	> 5s
Budget utilization	> 70%	> 90%
Heartbeat latency (p95)	> 500ms	> 2s
Audit buffer size	> 10,000 events	> 50,000 events
Policy sync age	> 5 minutes	> 15 minutes
Certificate expiry (outpost_certificate_expiry_seconds)	< 30 days	< 7 days

Example PromQL expressions for the outpost alert rules:

# Outpost heartbeat latency p95
- alert: OutpostHeartbeatLatencyHigh
  expr: histogram_quantile(0.95, rate(outpost_heartbeat_rtt_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

# Certificate expiry
- alert: OutpostCertificateExpiringSoon
  expr: outpost_certificate_expiry_seconds < 604800
  for: 1h
  labels:
    severity: critical

# Circuit breaker open
- alert: OutpostCircuitBreakerOpen
  expr: outpost_circuit_breaker_state == 1
  for: 2m
  labels:
    severity: critical

# Consecutive heartbeat failures
- alert: OutpostHeartbeatDown
  expr: outpost_heartbeat_consecutive_failures >= 3
  for: 1m
  labels:
    severity: critical

Caution

The BudgetUtilizationCritical alert fires per org_id. If you have many organizations, this alert can generate significant notification volume. Consider grouping by org_id in Alertmanager and routing to a budget-specific receiver rather than the default paging channel.

---

OpenTelemetry Integration

Platform OTel Configuration

The platform initializes the OTel SDK at startup and auto-instruments its major dependencies. Key environment variables:

Variable	Default	Description
OTEL_SERVICE_NAME	arbitex-platform	Service name reported in traces and metrics
OTEL_SERVICE_VERSION	settings.app_version	Service version; falls back to the application version setting
OTEL_EXPORTER_OTLP_ENDPOINT	—	gRPC OTLP collector endpoint (e.g., http://localhost:4317)
METRICS_ENABLED	true	Enable or disable the metrics middleware (true/false)

Auto-instrumented libraries on the platform:

• FastAPI — HTTP server spans
• httpx — outbound HTTP client spans (provider API calls)
• SQLAlchemy — database query spans
• Redis — cache operation spans

Log bridge: WARNING-level and above log records are forwarded to the OTLP endpoint. Trace ID and span ID are injected into log records via ContextVars for correlation.

Metric export interval: 60 seconds (batch export to OTLP collector).

Outpost OTel Configuration

The outpost OTel integration is opt-in and covers distributed tracing for the DLP pipeline.

Variable	Default	Description
OTEL_ENABLED	false	Enable OTel tracing (true to activate)
OTEL_EXPORTER_OTLP_ENDPOINT	—	gRPC OTLP collector endpoint

Tracer name: outpost.dlp

Auto-instrumented libraries on the outpost:

• FastAPI — HTTP server spans
• httpx — outbound mTLS calls to the platform

Trace Context Propagation

Arbitex uses W3C Trace Context for end-to-end distributed tracing across the platform and outpost boundary.

Component	Behavior
Platform (inbound)	Extracts traceparent from incoming HTTP requests via W3CTraceContextTextMapPropagator
Outpost (outbound)	Injects traceparent into outgoing mTLS calls to the platform
Composite propagator	TraceContext + W3CBaggage — both are propagated
Log correlation	trace_id and span_id are injected into ContextVars and appear in all structured log output

This means a single user request that enters through the outpost and proxies to the platform will produce a single connected trace in your APM tool, spanning both the outpost DLP pipeline and the platform policy evaluation.

Tip

If you use a third-party APM (Datadog, Honeycomb, Grafana Tempo), configure your OTLP collector to forward spans to your APM backend. Both the platform and outpost send to the same OTLP endpoint, so a single collector configuration covers both.

Full Environment Variable Reference

Variable	Component	Description
OTEL_EXPORTER_OTLP_ENDPOINT	Platform + Outpost	gRPC OTLP collector endpoint (e.g., http://otel-collector:4317)
OTEL_SERVICE_NAME	Platform	Service name tag on all telemetry (default: arbitex-platform)
OTEL_SERVICE_VERSION	Platform	Service version tag; falls back to settings.app_version
OTEL_ENABLED	Outpost	Enable OTel tracing on the outpost (true/false)
METRICS_ENABLED	Platform	Enable the HTTP metrics middleware (true/false, default: true)

---

Pushgateway Integration (Outpost)

The outpost supports pushing metrics to a Prometheus Pushgateway on a configurable interval. This is useful for short-lived outpost instances or deployments where inbound Prometheus scraping is not possible.

Grouping key format used by the outpost when pushing:

job/{job}/instance/{instance}/org_id/{org_id}

The Pushgateway integration includes a built-in circuit breaker to prevent push failures from consuming resources:

Parameter	Value
Failure threshold to open	5 consecutive failures
Recovery timeout	300 seconds
State after recovery timeout	Half-open (single probe attempt)

Caution

Pushgateway does not support metric expiry by default. If an outpost instance shuts down unexpectedly, its metrics remain in the Pushgateway until manually deleted. Use the --web.enable-admin-api flag on Pushgateway and delete stale grouping keys as part of your outpost shutdown automation.

Configure the Pushgateway endpoint in the outpost configuration. The outpost will fall back gracefully if the Pushgateway is unreachable; the circuit breaker prevents repeated failed push attempts from blocking request processing.