Admin day-2 operations

This guide covers day-2 operational procedures for Arbitex platform administrators. It assumes a running platform deployment. For initial setup, see Outpost Deployment and Kubernetes Deployment.

---

Registration gate

The allow_open_registration system configuration key controls whether new users can sign up without an invitation. The default is false — registration requires an invite token.

Value	Behavior
true	Anyone can register via the sign-up endpoint
false (default)	Registration returns 403 Forbidden unless the request includes a valid invite token

Configure the registration gate

# Check current registration setting
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/config/allow_open_registration" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Enable open registration (use with caution)
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/config/allow_open_registration" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": true}'

# Disable open registration (recommended for production)
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/config/allow_open_registration" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": false}'

This setting is stored in the database and takes effect immediately — no restart required.

Invite-based onboarding

When open registration is disabled, onboard users by sending invitations:

# Send an invitation
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/users/invite" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "[email protected]"
  }'

Invited users bypass the registration gate — the invite token is validated before the registration check.

Tip

For enterprise deployments using SAML SSO or SCIM provisioning, the registration gate is typically irrelevant — users are provisioned through the IdP. The gate is most important for deployments using local authentication.

---

Password policy

The platform enforces password requirements at registration, self-service password reset, and admin-initiated password reset.

Requirements

Rule	Value
Minimum length	12 characters
Complexity	At least one uppercase letter and one digit
Bcrypt byte cap	Passwords longer than 72 bytes are rejected (bcrypt limitation)
Breached password check	Rejected if found in HIBP database via k-anonymity API

Breached password detection

The platform checks passwords against the Have I Been Pwned Pwned Passwords API using the k-anonymity range protocol. Only the first 5 characters of the SHA-1 hash are sent to the HIBP service — the full password hash never leaves the platform. If the password appears in known data breaches, registration or reset is rejected with a message instructing the user to choose a different password.

Caution

The HIBP check requires outbound HTTPS access to api.pwnedpasswords.com. In air-gapped environments, the check is skipped gracefully — passwords are validated against length and complexity rules only.

Bcrypt migration (Outpost)

The Outpost automatically migrates admin keys from legacy SHA-256 hashing to bcrypt. When an admin authenticates with a SHA-256-hashed key, the Outpost:

1. Validates the password against the SHA-256 hash
2. Re-hashes the password with bcrypt
3. Stores the new bcrypt hash, replacing the SHA-256 hash
4. Logs a WARNING indicating the migration occurred

No operator action is required — the migration is transparent and happens on the next successful authentication.

---

Recovery rate limiting

The forgot-password and password-reset endpoints are protected by a Redis-aware sliding window rate limiter. This prevents credential stuffing and account enumeration through the recovery flow.

Rate limits

Endpoint	Limit	Window
POST /auth/forgot-password	Per email address	Sliding window
POST /auth/reset-password	Per token	Sliding window

Rate limit state is stored in Redis using a sorted set pipeline (ZADD/ZCARD), providing distributed sliding-window enforcement across all platform instances. When the limit is exceeded, the endpoint returns 429 Too Many Requests with a Retry-After header.

Note

The recovery rate limiter is separate from the general API rate limiter. Both apply independently — a user who has exhausted their recovery limit can still use other API endpoints normally.

---

Account recovery

Account recovery restores access for users who have lost all registered passkeys and all backup codes. The recovery flow issues a restricted token that permits only passkey enrollment, returning the account to a fully authenticated state once a new passkey is registered.

For the complete recovery flow documentation — including self-service recovery, token verification, OrgRecoveryPolicy, and security considerations — see the Account Recovery Guide.

Admin-initiated recovery

Admins can trigger recovery from Settings > Users > [user] > Initiate Recovery or via the API:

curl -s -X POST \
  "https://platform.example.com/api/v1/admin/users/${USER_ID}/recovery" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

A recovery email is sent to the user’s registered address. In development mode (no SMTP_HOST configured), the recovery token is logged to the server console (it is never returned in the API response for security reasons).

Recovery token lifecycle

Stage	Detail
Issued	JWT with type: "account_recovery", 1-hour expiry, unique JTI
Delivered	Email link (or API response in dev mode)
Verified	POST /api/auth/recovery/verify — validates type, expiry, JTI blacklist, active user
Consumed	JTI blacklisted (single-use). Restricted access token issued: 15-minute expiry, recovery_session=True
Resolved	User enrolls a new passkey. Full access restored immediately

The restricted session permits only passkey enrollment (/api/auth/webauthn/register/*), profile view (/api/auth/me), and logout (/api/auth/logout). All other endpoints return 403 recovery_session.

Caution

The 15-minute restricted token window is intentional. If the user cannot complete passkey enrollment in time, they must restart the recovery flow. This limits the attack surface of a stolen restricted token.

Recovery policy configuration

Each org has an OrgRecoveryPolicy controlling recovery behavior:

# View current policy
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/org/recovery-policy" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Enable self-service recovery
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/org/recovery-policy" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "admin_recovery_enabled": true,
    "self_service_recovery_enabled": true
  }'

Flag	Default	Description
admin_recovery_enabled	true	Allows admins to initiate recovery for users
self_service_recovery_enabled	false	Allows users to request recovery via email (rate-limited: 3/email/hour)

Passkey re-enrollment after recovery

After a successful recovery verification, the user’s restricted session allows only passkey enrollment:

1. The user’s browser calls POST /api/auth/webauthn/register/begin to get PublicKeyCredentialCreationOptions.
2. The browser invokes navigator.credentials.create() with the options.
3. The attestation response is sent to POST /api/auth/webauthn/register/complete.
4. On success, the restricted session is replaced with a standard access token — full access is restored.

Tip

Encourage users to register multiple passkeys (e.g., a hardware security key plus a platform authenticator) and store backup codes securely. Recovery is a last resort, not a routine workflow.

---

WebAuthn RP ID configuration

The WebAuthn Relying Party ID (WEBAUTHN_RP_ID) determines which domain passkeys are bound to. This is a critical deployment setting — changing it after users have enrolled passkeys will invalidate all existing credentials.

Environment variables

Variable	Default	Description
WEBAUTHN_RP_ID	localhost	Relying Party identifier — must match the domain users visit
WEBAUTHN_RP_ORIGIN	http://localhost:5173	Expected browser origin for ceremony verification

Production configuration

Set these variables to match your deployment domain:

# For an admin.arbitex.ai deployment
WEBAUTHN_RP_ID=admin.arbitex.ai
WEBAUTHN_RP_ORIGIN=https://admin.arbitex.ai

The RP ID must be an exact domain match or a registrable domain suffix. For example, if users visit admin.arbitex.ai, the RP ID can be admin.arbitex.ai or arbitex.ai (allowing passkeys to work across subdomains).

RP ID migration implications

Changing the RP ID after users have enrolled passkeys requires a coordinated migration:

1. Audit existing credentials: count active passkey registrations that will be affected.
2. Notify users: inform all passkey holders that re-enrollment will be required.
3. Update the RP ID: change WEBAUTHN_RP_ID and WEBAUTHN_RP_ORIGIN in the deployment configuration.
4. Trigger recovery for affected users: use admin-initiated recovery or self-service recovery to let users re-enroll passkeys against the new RP ID.
5. Clean up old credentials: after confirming re-enrollment, revoke orphaned credentials tied to the old RP ID.

Caution

There is no way to migrate WebAuthn credentials between RP IDs — the FIDO2 specification does not support it. Changing the RP ID is a destructive operation that requires full re-enrollment.

Challenge store

WebAuthn challenges are stored in Redis (webauthn:challenge:{key} pattern) with a 300-second TTL. On verification, challenges are consumed atomically via a GET + DELETE Redis pipeline to prevent replay attacks.

If Redis is unavailable, the platform falls back to an InMemoryChallengeStore — suitable for single-node deployments only. Multi-node deployments must use Redis to ensure challenges are accessible from any instance.

---

BYOK encryption (KMS configuration)

Organisations that need customer-managed encryption keys can configure a Key Management Service (KMS) provider. When enabled, audit log fields are encrypted using envelope encryption with the customer’s KMS key. See Security Architecture — BYOK for the technical design.

Configure BYOK via API

1. Choose a KMS provider (vault for Azure Key Vault, aws_kms for AWS KMS, or mock for testing).
2. Create the KMS configuration for your org.
3. Validate that the platform can reach and use the KMS.
4. BYOK encryption activates automatically on the next audit event.

# Create KMS configuration for an org
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/kms" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "kms_type": "vault",
    "endpoint": "https://myorg-vault.vault.azure.net",
    "key_id": "arbitex-audit-key",
    "auth_method": "managed_identity",
    "enabled": true
  }'

# Validate KMS connectivity (runs a test encrypt/decrypt cycle)
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/kms/validate" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# View current KMS configuration
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/kms" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Update KMS configuration
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/kms" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "kms_type": "vault",
    "endpoint": "https://myorg-vault.vault.azure.net",
    "key_id": "arbitex-audit-key-v2",
    "auth_method": "managed_identity",
    "enabled": true
  }'

# Remove KMS configuration (reverts to platform-managed encryption)
curl -s -X DELETE \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/kms" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

KMS providers

Provider	kms_type	Auth methods	Status
Azure Key Vault	vault	managed_identity, service_principal, token	Phase B
AWS KMS	aws_kms	IAM, access key	Phase B
Mock (testing)	mock	None	Available

Note

Azure Key Vault and AWS KMS providers are defined in the KMS abstraction layer but not yet implemented (Phase B). The mock provider is available for integration testing and development.

Key rotation

To rotate the KMS key, update the KMS configuration with the new key ID. The platform generates a new DEK on the next audit event using the new key. Existing encrypted audit entries remain readable using their stored wrapped DEKs — each entry records the KMS key reference used for wrapping.

Cloud portal

The Cloud portal provides a visual KMS configuration interface at Settings > Encryption (/portal/encryption). The portal supports provider selection, connection validation, and enable/disable toggling without using the API directly.

Caution

Removing a KMS configuration while encrypted audit entries exist does not decrypt them. The entries remain encrypted and will show placeholder text ("[encrypted - KMS unavailable]") in exports until the KMS is reconfigured.

---

Backup and restore

Arbitex supports full configuration export and import via the Cloud Portal API. Configuration backups capture your policy rules, DLP pipeline, and content filter settings so that you can restore a known-good state after an error or migrate settings between environments.

What is included

Category	Included
DLP rules and pipeline config	Yes
Policy rules and groups	Yes
Content filter configuration	Yes
Webhook definitions	Yes
Model routing configuration	Yes
Users and group memberships	No
Billing and subscription data	No
SSO / IdP configuration	No
Audit log history	No

SSO and billing data are managed by their respective providers and excluded by design. User accounts are restored via SCIM re-provisioning or IdP sync.

Export a configuration backup

# Export current org configuration
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/config/export" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Accept: application/json" \
  | jq . > arbitex-config-$(date +%Y%m%d-%H%M%S).json

The export response includes a schema_version field. Store this alongside the backup — the import endpoint validates schema compatibility before applying changes.

Import a configuration backup

Caution

Importing a configuration immediately overwrites the live policy rules and DLP settings for your org. Test imports in a staging environment before applying to production.

# Import a configuration backup
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/orgs/${ORG_ID}/config/import" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d @arbitex-config-20260314-120000.json

The import endpoint performs a dry-run validation before applying. If validation fails, the response returns a 400 with a structured error listing the conflicting fields. No changes are applied on failure.

Backup schedule recommendations

Trigger	Frequency	Retention
Automated daily backup	Every 24 hours	30 days
Pre-change manual backup	Before every policy or DLP change	90 days
Post-incident snapshot	After any incident resolution	1 year

Store backups in object storage (S3, Azure Blob, GCS) with server-side encryption. For disaster recovery, replicate backups to a second region.

For a full API reference, see Admin Operations API.

---

Certificate management

Arbitex uses TLS for all external connections, mTLS for Outpost-to-Platform communication, and Ed25519 signing keys for software updates and policy bundles.

TLS certificate rotation (Platform)

Caution

Rotating the platform TLS certificate causes a brief disruption to in-flight requests. Schedule rotations during low-traffic windows.

1. Generate or obtain the new certificate from your CA.
2. Upload it to your secret store (Kubernetes Secret, Azure Key Vault, AWS Secrets Manager).
3. Update the platform deployment to reference the new secret.
4. Perform a rolling restart: kubectl rollout restart deployment/arbitex-platform
5. Verify the new certificate is served: openssl s_client -connect platform.example.com:443 -brief 2>/dev/null | grep "Certificate chain"

mTLS certificate rotation (Outpost ↔ Platform)

The mTLS bundle is provisioned when an Outpost is registered. Certificates rotate automatically via the CertRotationClient — when a certificate is within 30 days of expiry, the Outpost requests renewal from the management plane, and the new certificate is installed atomically with zero downtime.

For manual reissuance (e.g., after key compromise), use the Cloud portal: Settings > Outposts > [Outpost] > Reissue Certificate. Copy the new outpost.pem and outpost.key to the certs/ directory — the Outpost detects and validates the new certificate on the next hourly check.

Certificate expiry monitoring

The platform exports Prometheus metrics for certificate expiry:

Metric	Description
tls_cert_expiry_seconds{cert="platform_tls"}	Seconds until the platform TLS certificate expires
tls_cert_expiry_seconds{cert="outpost_mtls"}	Seconds until the Outpost mTLS certificate expires
tls_cert_expiry_seconds{cert="ed25519_signing"}	Seconds until the Ed25519 key expires

Recommended alert thresholds:

# Prometheus alerting rule
- alert: CertificateExpiringSoon
  expr: tls_cert_expiry_seconds < 1209600  # 14 days
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate {{ $labels.cert }} expires in {{ $value | humanizeDuration }}"

- alert: CertificateExpiryCritical
  expr: tls_cert_expiry_seconds < 259200  # 3 days
  for: 10m
  labels:
    severity: critical

Rotation schedule

Certificate	Recommended interval	Emergency trigger
Platform TLS	90 days	Suspected private key exposure
Outpost mTLS	90 days	Outpost host compromise
Ed25519 signing key	12 months	Key material leak

---

Session management

Arbitex stores active user sessions in Redis. The session store is shared between all platform instances in a cluster.

Environment variables

Variable	Default	Description
SESSION_STORE_URL	—	Redis connection URL (redis://host:6379/0 or rediss:// for TLS)

Session parameters (concurrent limits, idle timeout, absolute timeout) are managed through the application configuration, not environment variables.

Force-terminate sessions for a user

Caution

Force-terminating sessions immediately invalidates all active tokens for that user. Any in-progress requests will fail with 401 Unauthorized.

# List active sessions for a user
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/sessions?user_id=${USER_ID}" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest"

# Terminate a specific session
curl -s -X DELETE \
  "https://platform.example.com/api/v1/admin/sessions/${SESSION_ID}" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest"

# Terminate ALL sessions for a user
curl -s -X DELETE \
  "https://platform.example.com/api/v1/admin/users/${USER_ID}/sessions" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest"

Session store health monitoring

The /readyz endpoint validates Redis connectivity as part of its readiness checks. The /health/deep endpoint provides detailed Redis status. Monitor Redis directly using standard Redis metrics (redis_connected_clients, redis_memory_used_bytes, redis_keyspace_hits_total).

# Check platform health including session store
curl -s https://platform.example.com/health/deep | jq '.checks.redis'

---

Audit log management

Arbitex maintains a tamper-evident audit log using an HMAC-SHA256 chain. Each log entry contains the HMAC of the previous entry, forming a linked chain that detects any modification, deletion, or reordering.

Retention policy configuration

Audit retention is managed through the audit_retention_days system configuration key:

# Get current retention setting
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/config/audit_retention_days" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest"

# Set retention to 365 days (1 year)
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/config/audit_retention_days" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Content-Type: application/json" \
  -d '{"value": 365}'

Parameter	Default	Maximum
audit_retention_days	90	3650 (10 years)

Export audit logs

JSON

# Export audit logs as JSON (date range)
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/audit/export" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "json",
    "start": "2026-03-01T00:00:00Z",
    "end": "2026-03-14T23:59:59Z"
  }' > audit-export-march.json

CSV

# Export audit logs as CSV
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/audit/export" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "csv",
    "start": "2026-03-01T00:00:00Z",
    "end": "2026-03-14T23:59:59Z"
  }' > audit-export-march.csv

Verify HMAC chain integrity

Walk the chain to detect any tampering. The platform provides a built-in verification endpoint:

# Verify integrity of all audit logs in a date range
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "Content-Type: application/json" \
  -d '{
    "start": "2026-03-01T00:00:00Z",
    "end": "2026-03-14T23:59:59Z"
  }' | jq '{valid, total_entries, errors}'

A healthy response returns "valid": true. If the chain is broken, errors contains details about the inconsistencies found.

To verify locally using an exported JSON file:

# Verify chain using local export (requires jq and openssl)
HMAC_KEY="${AUDIT_HMAC_KEY}"
PREV_HMAC=""
jq -c '.entries[]' audit-export-march.json | while IFS= read -r entry; do
  STORED_PREV=$(echo "$entry" | jq -r '.previous_hmac')
  if [ -n "$PREV_HMAC" ] && [ "$STORED_PREV" != "$PREV_HMAC" ]; then
    echo "CHAIN BREAK at entry: $(echo "$entry" | jq -r '.request_id')"
    exit 1
  fi
  PREV_HMAC=$(echo "$entry" | openssl dgst -sha256 -hmac "$HMAC_KEY" | awk '{print $2}')
done && echo "Chain intact."

Storage sizing estimates

Daily requests	JSON size/day	CSV size/day	90-day retention
10,000	~50 MB	~20 MB	~4.5 GB
100,000	~500 MB	~200 MB	~45 GB
1,000,000	~5 GB	~2 GB	~450 GB

For long-term retention, archive exported files to cold object storage. See the SIEM integration guide for streaming audit events to Splunk, Sentinel, or Elastic.

---

System health monitoring

Health check endpoints

Endpoint	Purpose	Auth required
GET /healthz	Kubernetes liveness probe — always returns 200 if process is alive	No
GET /readyz	Kubernetes readiness probe — validates DB and providers	No
GET /startup	Kubernetes startup probe — returns 200 after lifespan handler completes	No
GET /health/deep	Deep check — DB, Redis, secrets backend	No

# Readiness check with component breakdown
curl -s https://platform.example.com/readyz | jq .

Example response:

{
  "status": "ok",
  "checks": {
    "db": { "status": "ok" },
    "providers": { "status": "ok", "detail": "3 provider(s) registered" },
    "redis": { "status": "ok" },
    "geoip_enrichment": { "status": "ok", "detail": "ip2location=ok iptoasn=ok" }
  }
}

DB, providers, and Redis are critical — failure returns HTTP 503. GeoIP enrichment is advisory (degraded does not fail the check).

Key Prometheus metrics

Metric	Description
http_request_count	Total HTTP requests by method, path, status
http_request_latency_seconds	Request latency histogram
dlp_scan_latency_seconds	DLP pipeline scan latency by tier
policy_eval_time_seconds	Policy engine evaluation time
provider_response_time_seconds	Upstream AI provider response time
token_count_total	Token consumption by provider and model
budget_utilization_ratio	Per-org token budget utilization (0–1)
tls_cert_expiry_seconds	Certificate expiry countdown

The /metrics endpoint is unauthenticated and exposed on port 8080. Restrict access to the metrics port at the network level (firewall rules, Kubernetes NetworkPolicy).

Grafana dashboard references

Pre-built Grafana dashboards are available in two directories within the platform repository:

monitoring/grafana/dashboards/ — operational dashboards:

Dashboard	File	Purpose
Health Overview	health-overview.json	System health, request rates, latency, error rates
Usage Metering	usage-metering.json	Token usage, provider distribution, top consumers
Compliance	compliance.json	DLP scan rates, detection categories, audit chain status
DLP Pipeline	dlp-pipeline.json	Scan latency by tier, block rates, NER/DeBERTa utilization
Provider Performance	provider-performance.json	Provider response times, error rates, circuit breaker status
Security Events	security-events.json	Auth failures, DLP blocks, anomaly detections

tools/grafana/dashboards/ — infrastructure dashboards:

Dashboard	File	Purpose
DLP Pipeline	arbitex-dlp-pipeline.json	DLP pipeline performance and throughput
LLM Providers	arbitex-llm-providers.json	Provider latency, availability, cost tracking
Observability	arbitex-observability.json	OTel traces, log volume, metric cardinality
Infrastructure	arbitex-infrastructure.json	Container health, resource utilization, network metrics

Import dashboards via Grafana UI (Dashboards → Import → Upload JSON) or the Grafana provisioning API. For full OTel configuration including OTLP export, see the OTel configuration guide.

Alert thresholds

Alert	Condition	Severity
High error rate	http_error_count rate > 1% over 5 min	Warning
Database unavailable	components.database.status != "ok"	Critical
DLP latency spike	dlp_scan_latency_seconds p99 > 2s	Warning
Budget near limit	budget_utilization_ratio > 0.9	Warning
Certificate expiry	tls_cert_expiry_seconds < 1209600	Warning

---

Kill switch and emergency controls

Danger

All kill switch operations are irreversible through the API — they require an explicit re-enable call. All activations are written to the audit log with the activating admin’s identity and timestamp.

Provider kill switch

Disable a specific AI provider without affecting other providers. The kill switch operates per-provider by name (not UUID):

# Disable a provider
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/kill-switch/providers/${PROVIDER_NAME}/disable" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest" \
  -d '{"reason": "Provider incident — see status.anthropic.com"}'

# Re-enable
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/kill-switch/providers/${PROVIDER_NAME}/enable" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "X-Requested-With: XMLHttpRequest"

When a provider is disabled, requests routing to that provider receive a 503 with provider_unavailable error code. Fallback routing (if configured) activates automatically.

Per-model kill switches are also available at /api/v1/admin/kill-switch/models/{model}/disable and /api/v1/admin/kill-switch/models/{model}/enable.

Emergency runbook format

For each emergency scenario, maintain a runbook with this structure:

## Runbook: [Scenario Name]
**Trigger**: [What causes this scenario]
**Impact**: [What services/users are affected]
**Severity**: P1 / P2 / P3

### Immediate actions (< 5 min)
1. ...

### Investigation steps
1. ...

### Resolution steps
1. ...

### Post-incident
- [ ] Re-enable affected controls
- [ ] Verify audit log chain integrity
- [ ] Write incident retrospective
- [ ] Update runbook if needed

---

DLP NER gate configuration

The NER gate controls whether Tier 2 NER (Named Entity Recognition) runs during DLP scanning. NER detects person names, organisations, locations, and other natural-language entities that structural regex patterns miss — but it adds 10–50 ms of latency per scan. The gate lets administrators skip NER for orgs that don’t need it.

See DLP Pipeline Architecture — NER gate for the technical design.

DLP_NER_FORCE system config key

The DLP_NER_FORCE key in system_config controls NER behavior globally. Per-org overrides use the key dlp_ner_force:{org_id}.

Value	Behavior
auto (default)	NER runs only when the org’s enabled compliance packs require it AND trigger keywords are present in the text
always	NER always runs regardless of pack configuration
never	NER is never invoked — only regex (Tier 1) and DeBERTa (Tier 3) run

# Check current NER force setting
curl -s -X GET \
  "https://platform.example.com/api/v1/admin/config/dlp_ner_force" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Set NER force to "always" (for orgs requiring maximum detection)
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/config/dlp_ner_force" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": "always"}'

# Set per-org NER force override
curl -s -X PUT \
  "https://platform.example.com/api/v1/admin/config/dlp_ner_force:${ORG_ID}" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"value": "never"}'

When to adjust

Scenario	Recommended setting
Org handles HIPAA/GDPR/CCPA data with person names	auto (default) — packs flag NER as required
Org uses PCI-DSS/SOX only (structured patterns)	auto — NER is skipped automatically
Maximum detection sensitivity regardless of latency	always
Ultra-low-latency scanning (code-only workloads)	never — Tier 1 regex only

Note

The auto mode uses a keyword pre-filter (~57 NER-trigger keywords like “patient”, “address”, “diagnosis”) to skip NER on text that clearly contains no entity-like content. This means even with NER-requiring packs enabled, a code snippet with no NER keywords will skip the NER call.

Compiled Rust scanner backend

The DLP_SCANNER_BACKEND environment variable selects the Tier 1 (regex) scanning engine. Set to rust to use the compiled Rust scanner for higher throughput, or leave at the default python for the standard regex engine.

# Enable Rust scanner backend (in .env or environment)
DLP_SCANNER_BACKEND=rust

The Rust backend falls back to Python transparently if the compiled extension is not installed. Both the platform and Outpost support the same flag — the compiled scanner is identical in both deployments.

---

---

Email DLP administration (Cloud Portal)

The Cloud Portal provides a visual interface for email DLP quarantine management, accessible at Settings > Email DLP.

Quarantine dashboard

The quarantine dashboard (/portal/email-dlp/quarantine) displays quarantined emails with filtering by date range, sender, recipient, and DLP entity type. Admins can release or permanently delete quarantined messages directly from the dashboard.

Statistics page

The stats page (/portal/email-dlp/stats) shows email DLP scan volume, detection rates by entity type, quarantine queue depth, and false-positive rates over configurable time windows.

Configuration page

The config page (/portal/email-dlp/config) allows admins to configure email relay settings, DLP scan sensitivity, and quarantine retention policies without using the API directly.

The Cloud Portal proxies these operations through 8 admin API routes to the platform backend. See the Email DLP Guide for the full API reference.

---

Avatar storage

User avatars are stored as content-addressed WebP files with org-scoped directories for tenant isolation.

Storage pipeline

When a user uploads an avatar:

1. Validate: check file size (max 2 MB), MIME type (PNG, JPEG, WebP, AVIF), and magic bytes
2. Optimize: Pillow resizes and crops to 256x256, converts to WebP (quality 80, method 6)
3. Hash: SHA-256 of the optimized WebP data, truncated to 16 hex characters
4. Store: write to {AVATAR_UPLOAD_DIR}/{org_id}/{hash}.webp
5. Clean up: delete the previous avatar file if the hash differs

Content-addressed filenames provide automatic cache-busting — a re-upload with a different image produces a different filename, while re-uploading the same image is a no-op.

Configuration

Variable	Default	Description
AVATAR_UPLOAD_DIR	/app/uploads/avatars	Directory for avatar storage (Docker volume mount)
UPLOAD_DIR	/app/uploads	Parent directory for all uploads (set in Docker Compose)

In Docker Compose, the upload directory is configured as a named volume:

services:
  backend:
    environment:
      UPLOAD_DIR: /app/uploads
    volumes:
      - uploads:/app/uploads

IdP photo sync

When users are provisioned via SCIM, the platform syncs profile photos from the identity provider. The SCIM service extracts the primary photo URL from the photos array and stores it in profile_settings.avatar_url with avatar_source: "idp".

User-uploaded avatars take precedence: if a user has manually uploaded an avatar (avatar_source: "upload"), the IdP photo is not overwritten during SCIM updates. This ensures that user customizations are preserved across IdP sync cycles.

Orphan cleanup

The POST /api/v1/admin/avatars/cleanup endpoint scans avatar directories for files not referenced by any user in the database. Orphans can accumulate when users are deleted or when avatar uploads fail after writing to disk.

# Scan and remove orphaned avatar files (admin only)
curl -s -X POST \
  "https://platform.example.com/api/v1/admin/avatars/cleanup" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

The scan can be scoped to a single org or run across all orgs (superadmin). Deleted filenames are returned in the response for audit purposes.

---

E2E browser validation

The platform includes end-to-end Playwright test suites that validate admin UI workflows in a real browser. These specs cover:

• Admin login and session management
• User invitation and role assignment
• DLP configuration and NER gate toggling
• Audit log export and chain verification
• Kill switch activation and provider management
• Email DLP quarantine review workflows

Run the admin E2E specs during deployment validation:

npx playwright test tests/admin/ --project=chromium

The specs use isolated test databases and do not affect production data.

---

Tokenizer serialization

The DLP pipeline’s ML services (AppGuard classifier and DeBERTa validator) use standalone tokenizer files instead of the full transformers library at runtime. This reduces Docker image size by approximately 2 GB by eliminating the torch and transformers dependencies from runtime images.

How it works

A build-time script (scripts/serialize_tokenizers.py) exports HuggingFace tokenizers to standalone tokenizer.json files using the tokenizers Rust library’s backend_tokenizer.save() method. The exported files are verified for equivalence against the original AutoTokenizer output.

At runtime, services load tokenizers via tokenizers.Tokenizer.from_file("tokenizer.json") instead of transformers.AutoTokenizer.from_pretrained(). The DeBERTa validator supports a dual-path strategy — it prefers the standalone tokenizer but falls back to AutoTokenizer if the standalone file is not available.

Impact on Docker images

Component	Before	After	Savings
AppGuard classifier	transformers + torch	tokenizers only	~1.5 GB
DeBERTa validator	transformers + torch (runtime)	tokenizers + onnxruntime + numpy	~1.5 GB

The Dockerfiles use multi-stage builds: the builder stage installs torch and transformers for tokenizer serialization, while the runtime stage installs only the lean requirements-runtime.txt.