Observability API

API reference for the Observability suite — drift detection, bias & fairness monitoring, and model evaluation intelligence. These endpoints power the Epic V observability dashboard and are organized across three router domains.

Base paths:

/api/v1/observability/drift — Drift & anomaly detection
/api/v1/observability/fairness — Bias & fairness analysis
/api/v1/observability/eval — Model evaluation & analytics

Authentication: Authorization: Bearer <admin-jwt> on all requests. Admin role required.

Drift Detection

Epic V Phase 3. Drift detection endpoints compare recent activity against a configurable baseline window and flag statistically significant deviations. All three endpoints share a common parameter pattern for baseline and comparison windows.

GET /api/v1/observability/drift/latency

Detect latency drift per model and provider using z-score analysis. Compares the current window latency distribution against a baseline window and flags anomalous deviations.

Query Parameters

Parameter	Type	Default	Description
`model_id`	string	—	Filter results to a specific model. Optional.
`provider`	string	—	Filter results to a specific provider. Optional.
`baseline_days`	integer	`7`	Baseline window in days (1–90).
`comparison_minutes`	integer	`60`	Current comparison window in minutes (1–1440).
`anomaly_threshold`	float	`2.0`	Z-score threshold for flagging anomalies (0.5–10.0).

Response

{
  "results": [
    {
      "model_id": "gpt-4o",
      "provider": "openai",
      "baseline_avg_ms": 342.5,
      "current_avg_ms": 891.2,
      "z_score": 3.41,
      "is_anomalous": true
    },
    {
      "model_id": "claude-3-5-sonnet",
      "provider": "anthropic",
      "baseline_avg_ms": 285.1,
      "current_avg_ms": 301.8,
      "z_score": 0.42,
      "is_anomalous": false
    }
  ],
  "baseline_days": 7,
  "comparison_minutes": 60,
  "anomaly_threshold": 2.0
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/drift/latency \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "model_id=gpt-4o" \
  --data-urlencode "baseline_days=14" \
  --data-urlencode "anomaly_threshold=3.0"

GET /api/v1/observability/drift/topic

Detect topic distribution drift per tenant using Jensen-Shannon divergence. Compares the current topic category distribution against a baseline period.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.
`baseline_days`	integer	`7`	Baseline window in days (1–90).
`comparison_minutes`	integer	`60`	Current comparison window in minutes (1–1440).
`anomaly_threshold`	float	`0.3`	Jensen-Shannon divergence threshold for flagging drift (0.01–1.0).

Response

{
  "results": [
    {
      "tenant_id": "acme-corp",
      "baseline_distribution": {
        "code_generation": 0.45,
        "summarization": 0.30,
        "translation": 0.15,
        "other": 0.10
      },
      "current_distribution": {
        "code_generation": 0.20,
        "summarization": 0.15,
        "translation": 0.05,
        "data_extraction": 0.60
      },
      "jsd_score": 0.52,
      "is_anomalous": true
    }
  ],
  "baseline_days": 7,
  "comparison_minutes": 60,
  "anomaly_threshold": 0.3
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/drift/topic \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "tenant_id=acme-corp" \
  --data-urlencode "baseline_days=30"

GET /api/v1/observability/drift/behavior

Detect anomalous user behavior patterns including volume spikes, unusual time-of-day activity, and prompt length deviations. Computes a composite anomaly score from multiple behavioral signals.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.
`user_id`	string	—	Filter to a specific user. Optional.
`baseline_days`	integer	`7`	Baseline window in days (1–90).
`comparison_minutes`	integer	`60`	Current comparison window in minutes (1–1440).
`sensitivity`	string	`medium`	Detection sensitivity. One of `low` (3σ), `medium` (2σ), `high` (1.5σ).

Response

{
  "results": [
    {
      "user_id": "usr_a1b2c3",
      "tenant_id": "acme-corp",
      "anomaly_signals": {
        "volume_spike": true,
        "unusual_time": false,
        "prompt_length_deviation": true
      },
      "composite_score": 4.2,
      "is_anomalous": true
    }
  ],
  "baseline_days": 7,
  "comparison_minutes": 60,
  "sensitivity": "medium"
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/drift/behavior \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "sensitivity=high" \
  --data-urlencode "comparison_minutes=120"

Bias & Fairness

Epic V Phase 4. Fairness endpoints analyze LLM response quality and sentiment across user groups defined by tenant metadata attributes (e.g., department, role). These endpoints support EU AI Act and NIST AI RMF fairness monitoring requirements.

GET /api/v1/observability/fairness/disparate-treatment

Detect disparate treatment across user groups by comparing response quality metrics using statistical tests. Returns per-attribute analysis with significance flags.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.
`group_attributes`	string	`department,role`	Comma-separated list of user metadata attributes to compare.
`baseline_days`	integer	`7`	Analysis window in days (1–90).

Response

{
  "results": [
    {
      "attribute": "department",
      "comparisons": [
        {
          "group_a": "engineering",
          "group_b": "marketing",
          "metric": "response_quality",
          "group_a_mean": 0.87,
          "group_b_mean": 0.71,
          "p_value": 0.003,
          "is_significant": true
        }
      ]
    },
    {
      "attribute": "role",
      "comparisons": [
        {
          "group_a": "manager",
          "group_b": "individual_contributor",
          "metric": "response_quality",
          "group_a_mean": 0.84,
          "group_b_mean": 0.82,
          "p_value": 0.41,
          "is_significant": false
        }
      ]
    }
  ],
  "tenant_id": "acme-corp",
  "baseline_days": 7,
  "total_violations": 1
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/fairness/disparate-treatment \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "group_attributes=department,role,location" \
  --data-urlencode "baseline_days=30"

GET /api/v1/observability/fairness/sentiment-bias

Detect sentiment bias in LLM responses by group or topic. Flags when sentiment for a specific group deviates significantly from the population baseline.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.
`group_attributes`	string	`department,role`	Comma-separated list of user metadata attributes to compare.
`baseline_days`	integer	`7`	Analysis window in days (1–90).
`sensitivity_threshold`	float	`0.15`	Deviation threshold for flagging bias (0.01–1.0).

Response

{
  "results": [
    {
      "group": "engineering",
      "attribute": "department",
      "avg_sentiment": 0.72,
      "population_avg": 0.68,
      "deviation": 0.04,
      "is_biased": false
    },
    {
      "group": "support",
      "attribute": "department",
      "avg_sentiment": 0.45,
      "population_avg": 0.68,
      "deviation": -0.23,
      "is_biased": true
    }
  ],
  "tenant_id": "acme-corp",
  "baseline_days": 7,
  "sensitivity_threshold": 0.15,
  "total_biased": 1
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/fairness/sentiment-bias \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "sensitivity_threshold=0.10" \
  --data-urlencode "baseline_days=14"

GET /api/v1/observability/fairness/summary

Aggregate all fairness signals into a dashboard-ready summary. Returns active fairness flags, trend data, an overall fairness score (0–100), and actionable recommendations.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.
`time_range_days`	integer	`7`	Analysis window in days (1–90).

Response

{
  "overall_score": 82,
  "active_flags": 2,
  "trend": "improving",
  "disparate_treatment": {
    "total_violations": 1,
    "top_attribute": "department"
  },
  "sentiment_bias": {
    "total_biased": 1,
    "top_group": "support"
  },
  "recommendations": [
    "Review response patterns for support department — sentiment deviation exceeds threshold",
    "Consider adjusting model routing for affected groups"
  ],
  "tenant_id": "acme-corp",
  "time_range_days": 7
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/fairness/summary \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "time_range_days=30"

Model Evaluation

Epic V Phase 5. Evaluation endpoints provide model comparison, golden set regression testing, and analytics dashboards. These power the model intelligence layer of the observability suite.

POST /api/v1/observability/eval/comparisons

Create and immediately execute a model comparison. Sends each prompt in the set to all specified models and returns per-model performance statistics.

Request Body

Field	Type	Required	Description
`prompt_set`	array of strings	Yes	Prompts to evaluate (1–50 items).
`models`	array of strings	Yes	Model identifiers in `provider/model_id` format (minimum 2).

{
  "prompt_set": [
    "Explain quantum computing in simple terms",
    "Write a Python function to sort a list",
    "Summarize the key points of GDPR Article 17"
  ],
  "models": [
    "openai/gpt-4o",
    "anthropic/claude-3-5-sonnet"
  ]
}

Response (201 Created)

{
  "id": "cmp_8f3a1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
  "tenant_id": "acme-corp",
  "prompt_count": 3,
  "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"],
  "results": [
    {
      "model": "openai/gpt-4o",
      "avg_latency_ms": 1240,
      "avg_tokens": 187,
      "avg_quality_score": 0.84
    },
    {
      "model": "anthropic/claude-3-5-sonnet",
      "avg_latency_ms": 980,
      "avg_tokens": 203,
      "avg_quality_score": 0.91
    }
  ],
  "created_at": "2026-04-05T14:30:00Z"
}

Example

curl -X POST https://api.arbitex.ai/api/v1/observability/eval/comparisons \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_set": ["Explain quantum computing in simple terms"],
    "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"]
  }'

GET /api/v1/observability/eval/comparisons

List model comparisons for the current tenant with pagination.

Query Parameters

Parameter	Type	Default	Description
`page`	integer	`1`	Page number (1-based).
`page_size`	integer	`20`	Results per page (1–100).

Response

{
  "comparisons": [
    {
      "id": "cmp_8f3a1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
      "tenant_id": "acme-corp",
      "prompt_count": 3,
      "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"],
      "created_at": "2026-04-05T14:30:00Z"
    }
  ],
  "total": 1
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/comparisons \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "page=1" \
  --data-urlencode "page_size=10"

GET /api/v1/observability/eval/comparisons/{comparison_id}

Retrieve a single model comparison by UUID, including full per-model results.

Path Parameters

Parameter	Type	Description
`comparison_id`	string (UUID)	The comparison identifier.

Response

{
  "id": "cmp_8f3a1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
  "tenant_id": "acme-corp",
  "prompt_count": 3,
  "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"],
  "results": [
    {
      "model": "openai/gpt-4o",
      "avg_latency_ms": 1240,
      "avg_tokens": 187,
      "avg_quality_score": 0.84
    },
    {
      "model": "anthropic/claude-3-5-sonnet",
      "avg_latency_ms": 980,
      "avg_tokens": 203,
      "avg_quality_score": 0.91
    }
  ],
  "created_at": "2026-04-05T14:30:00Z"
}

Example

curl https://api.arbitex.ai/api/v1/observability/eval/comparisons/cmp_8f3a1b2c-4d5e-6f7a-8b9c-0d1e2f3a4b5c \
  -H "Authorization: Bearer $ADMIN_TOKEN"

POST /api/v1/observability/eval/golden-sets

Create a new golden set for the current tenant. A golden set is a collection of prompt/expected-output pairs used for regression testing against models.

Request Body

Field	Type	Required	Description
`name`	string	Yes	Name for the golden set.
`description`	string	No	Optional description.

{
  "name": "Customer Support Quality",
  "description": "Regression tests for support-facing prompts"
}

Response (201 Created)

{
  "id": "gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "tenant_id": "acme-corp",
  "name": "Customer Support Quality",
  "description": "Regression tests for support-facing prompts",
  "created_at": "2026-04-05T15:00:00Z",
  "updated_at": "2026-04-05T15:00:00Z"
}

Example

curl -X POST https://api.arbitex.ai/api/v1/observability/eval/golden-sets \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Customer Support Quality", "description": "Regression tests for support-facing prompts"}'

GET /api/v1/observability/eval/golden-sets

List golden sets for the current tenant with pagination.

Query Parameters

Parameter	Type	Default	Description
`page`	integer	`1`	Page number (1-based).
`page_size`	integer	`20`	Results per page (1–100).

Response

{
  "golden_sets": [
    {
      "id": "gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "tenant_id": "acme-corp",
      "name": "Customer Support Quality",
      "description": "Regression tests for support-facing prompts",
      "created_at": "2026-04-05T15:00:00Z",
      "updated_at": "2026-04-05T15:00:00Z"
    }
  ],
  "total": 1
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/golden-sets \
  -H "Authorization: Bearer $ADMIN_TOKEN"

GET /api/v1/observability/eval/golden-sets/{golden_set_id}

Retrieve a golden set with all its test items.

Path Parameters

Parameter	Type	Description
`golden_set_id`	string (UUID)	The golden set identifier.

Response

{
  "id": "gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "tenant_id": "acme-corp",
  "name": "Customer Support Quality",
  "description": "Regression tests for support-facing prompts",
  "items": [
    {
      "id": "gsi_a1b2c3d4",
      "prompt": "How do I reset my password?",
      "expected_output": "Navigate to Settings > Security > Change Password",
      "metadata": {"category": "account"},
      "scoring_rubric": "Must mention Settings and Security paths"
    }
  ],
  "created_at": "2026-04-05T15:00:00Z",
  "updated_at": "2026-04-05T15:05:00Z"
}

Example

curl https://api.arbitex.ai/api/v1/observability/eval/golden-sets/gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d \
  -H "Authorization: Bearer $ADMIN_TOKEN"

DELETE /api/v1/observability/eval/golden-sets/{golden_set_id}

Delete a golden set and all its items and evaluation runs. This action is irreversible.

Path Parameters

Parameter	Type	Description
`golden_set_id`	string (UUID)	The golden set identifier.

Response

Returns 204 No Content on success.

Example

curl -X DELETE https://api.arbitex.ai/api/v1/observability/eval/golden-sets/gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d \
  -H "Authorization: Bearer $ADMIN_TOKEN"

POST /api/v1/observability/eval/golden-sets/{golden_set_id}/items

Add a test item to an existing golden set.

Path Parameters

Parameter	Type	Description
`golden_set_id`	string (UUID)	The target golden set identifier.

Request Body

Field	Type	Required	Description
`prompt`	string	Yes	The test prompt.
`expected_output`	string	Yes	The expected model response.
`metadata`	object	No	Arbitrary key-value metadata for categorization.
`scoring_rubric`	string	No	Free-text rubric for scoring the response.

{
  "prompt": "What is our refund policy?",
  "expected_output": "Full refund within 30 days of purchase",
  "metadata": {"category": "billing"},
  "scoring_rubric": "Must state 30-day window and full refund"
}

Response (201 Created)

{
  "id": "gsi_b2c3d4e5",
  "prompt": "What is our refund policy?",
  "expected_output": "Full refund within 30 days of purchase",
  "metadata": {"category": "billing"},
  "scoring_rubric": "Must state 30-day window and full refund"
}

Example

curl -X POST https://api.arbitex.ai/api/v1/observability/eval/golden-sets/gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d/items \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What is our refund policy?",
    "expected_output": "Full refund within 30 days of purchase",
    "scoring_rubric": "Must state 30-day window and full refund"
  }'

POST /api/v1/observability/eval/golden-sets/{golden_set_id}/run

Run a golden set evaluation against a specified model. Sends each prompt to the model, scores responses using per-item rubrics, and detects regression against the rolling average of the last 5 runs for the same model.

Path Parameters

Parameter	Type	Description
`golden_set_id`	string (UUID)	The golden set to evaluate.

Query Parameters

Parameter	Type	Required	Description
`model`	string	Yes	Model identifier in `provider/model_id` format (e.g., `openai/gpt-4o`).

Response

{
  "id": "run_c3d4e5f6",
  "golden_set_id": "gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "model": "openai/gpt-4o",
  "overall_score": 0.85,
  "item_scores": [
    {
      "item_id": "gsi_a1b2c3d4",
      "score": 0.90,
      "model_response": "Go to Settings > Security > Change Password to reset your password."
    },
    {
      "item_id": "gsi_b2c3d4e5",
      "score": 0.80,
      "model_response": "You can get a full refund within 30 days."
    }
  ],
  "rolling_avg": 0.88,
  "regression_detected": true,
  "created_at": "2026-04-05T16:00:00Z"
}

The regression_detected flag is true when the current overall_score drops below the rolling_avg of the last 5 runs for the same model and golden set.

Example

curl -X POST "https://api.arbitex.ai/api/v1/observability/eval/golden-sets/gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d/run?model=openai/gpt-4o" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

GET /api/v1/observability/eval/golden-sets/{golden_set_id}/runs

List all evaluation runs for a golden set, newest first.

Path Parameters

Parameter	Type	Description
`golden_set_id`	string (UUID)	The golden set identifier.

Query Parameters

Parameter	Type	Default	Description
`page`	integer	`1`	Page number (1-based).
`page_size`	integer	`20`	Results per page (1–100).

Response

{
  "runs": [
    {
      "id": "run_c3d4e5f6",
      "golden_set_id": "gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "model": "openai/gpt-4o",
      "overall_score": 0.85,
      "regression_detected": true,
      "created_at": "2026-04-05T16:00:00Z"
    }
  ],
  "total": 1
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/golden-sets/gs_1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d/runs \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "page_size=50"

Analytics

The analytics endpoints aggregate signals across all Epic V phases into dashboard-ready summaries and time-series data.

GET /api/v1/observability/eval/analytics/summary

Full observability dashboard payload aggregating signals across all Epic V phases — drift, fairness, and evaluation data in a single response.

Query Parameters

Parameter	Type	Default	Description
`days`	integer	`30`	Lookback window in days (1–365).
`tenant_id`	string	—	Filter to a specific tenant. Optional.

Response

{
  "period_days": 30,
  "drift": {
    "latency_anomalies": 2,
    "topic_drifts": 1,
    "behavior_anomalies": 3
  },
  "fairness": {
    "overall_score": 82,
    "active_flags": 2,
    "trend": "improving"
  },
  "evaluation": {
    "total_comparisons": 14,
    "total_golden_set_runs": 38,
    "regressions_detected": 3
  },
  "health_score": 76,
  "tenant_id": "acme-corp"
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/analytics/summary \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "days=90"

GET /api/v1/observability/eval/analytics/trends

Time-series trend data for observability signals over a configurable window. Supports optional filtering by signal type.

Query Parameters

Parameter	Type	Default	Description
`days`	integer	`30`	Lookback window in days (1–365).
`signal_filter`	string	—	Filter to a specific signal (e.g., `latency_drift`, `fairness`, `regression`). Optional.
`tenant_id`	string	—	Filter to a specific tenant. Optional.

Response

{
  "period_days": 30,
  "data_points": [
    {
      "date": "2026-04-01",
      "latency_anomalies": 0,
      "topic_drifts": 0,
      "fairness_score": 85,
      "regressions": 0
    },
    {
      "date": "2026-04-02",
      "latency_anomalies": 1,
      "topic_drifts": 0,
      "fairness_score": 83,
      "regressions": 1
    }
  ],
  "signal_filter": null,
  "tenant_id": "acme-corp"
}

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/analytics/trends \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  --data-urlencode "days=60" \
  --data-urlencode "signal_filter=latency_drift"

GET /api/v1/observability/eval/analytics/health

Lightweight health score endpoint for quick dashboard status checks. Returns a single 0–100 composite score derived from all observability signals.

Query Parameters

Parameter	Type	Default	Description
`tenant_id`	string	—	Filter to a specific tenant. Optional.

Response

{
  "health_score": 76,
  "status": "warning",
  "components": {
    "drift": "healthy",
    "fairness": "warning",
    "evaluation": "healthy"
  },
  "tenant_id": "acme-corp"
}

The status field maps to score ranges: healthy (80–100), warning (50–79), critical (0–49).

Example

curl -G https://api.arbitex.ai/api/v1/observability/eval/analytics/health \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Error Codes

Status	Code	Description
400	`validation_error`	Invalid query parameter value (e.g., `baseline_days` out of range, `sensitivity` not one of `low`/`medium`/`high`, empty `prompt_set`, fewer than 2 models).
400	`empty_golden_set`	Attempted to run a golden set with no items.
401	`unauthorized`	Missing or invalid `Authorization` header.
403	`forbidden`	Valid token but insufficient privileges — admin role required.
404	`not_found`	Comparison or golden set UUID does not exist.

Error Response Shape

{
  "detail": "Golden set not found"
}

API Reference Index — full list of admin API endpoints
Output Quality API — quality metrics, alerts, and model comparison scoring