API Key Security Best Practices

API Key Security Best Practices

Arbitex uses API keys for two distinct purposes: authenticating portal users and services to the Cloud Platform, and authenticating client applications to the Outpost proxy. Each key type has different creation, storage, and rotation requirements.

This guide covers both key types, secure storage practices, rotation procedures, monitoring, and incident response for key compromise.

---

Key types

Arbitex has two independent API key systems:

Cloud portal API keys (arb_ prefix)

Cloud portal keys authenticate admin and service access to the Cloud Platform API. They are org-scoped, created through the portal or API, and stored as SHA-256 hashes.

Property	Value
Prefix	arb_
Format	arb_ + base64url(32 random bytes) ≈ 47 characters total
Hash storage	SHA-256 hex digest (64 characters)
Scope	Organization (org_id)
Revocation	Soft-delete (sets revoked_at timestamp — audit trail preserved)
Display	key_prefix shows first 8 characters after arb_ for identification
Created by	Email of the creating user is recorded

Platform user API keys (mmx_ prefix)

Platform-level user keys authenticate direct API access for individual users within the platform backend. These are typically used for CI/CD pipelines or service integrations.

Property	Value
Prefix	mmx_
Format	mmx_ + 32 hex characters = 36 characters total
Hash storage	SHA-256 hex digest (64 characters)
Scope	User (user_id) within a tenant
Revocation	Hard-delete (row removed from database)
Display	key_prefix shows first 8 characters (includes mmx_ prefix)

Outpost inbound authentication (OUTPOST_API_KEY)

The outpost proxy accepts an API key via Authorization: Bearer <key> for client authentication to the /v1/chat/completions endpoint:

Property	Value
Env var	OUTPOST_API_KEY
Format	Operator-defined string (no prefix requirement)
Storage	Environment variable or Kubernetes Secret
Scope	Single outpost instance

Development mode

When OUTPOST_API_KEY is empty, the outpost skips inbound authentication entirely. This is intended for local development only — never deploy to production without setting this value.

Outpost → Platform authentication (mTLS)

The outpost authenticates to the platform management plane using mTLS client certificates, not API keys. The relevant configuration is:

Env var	Description
OUTPOST_CERT_PATH	Client certificate (default: certs/outpost.pem)
OUTPOST_KEY_PATH	Private key (default: certs/outpost.key)
OUTPOST_CA_PATH	CA certificate (default: certs/ca.pem)

---

Key creation

Cloud portal (arb_ keys)

Via portal UI:

1. Navigate to Settings → API Keys in the Arbitex portal.
2. Click Create API Key.
3. Enter a descriptive name (max 128 characters).
4. Optionally set an expiration period: 30, 60, 90, or 365 days.
5. Optionally configure scopes.
6. Copy the plaintext key immediately — it is displayed once and cannot be retrieved again.

Via API:

curl -X POST https://api.arbitex.ai/v1/orgs/{org_id}/api-keys \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "CI Pipeline - Production",
    "expires_in_days": 90,
    "scopes": []
  }'

Response (201 Created):

{
  "key": {
    "id": "a1b2c3d4-...",
    "org_id": "e5f6a7b8-...",
    "name": "CI Pipeline - Production",
    "key_prefix": "x9y8z7w6",
    "scopes": [],
    "created_by": "[email protected]",
    "last_used_at": null,
    "expires_at": "2026-06-15T00:00:00Z",
    "revoked_at": null,
    "created_at": "2026-03-15T12:00:00Z"
  },
  "plaintext_key": "arb_x9y8z7w6..."
}

One-time display

The plaintext_key value is returned only in the creation response. Arbitex stores only the SHA-256 hash — the plaintext cannot be recovered. If you lose the key, revoke it and create a new one.

Platform user keys (mmx_ keys)

curl -X POST https://api.arbitex.ai/api/auth/api-keys \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Deployment automation",
    "expires_in_days": 90
  }'

Response (201 Created):

{
  "id": "a1b2c3d4-...",
  "key_prefix": "mmx_a1b2",
  "description": "Deployment automation",
  "created_at": "2026-03-15T12:00:00Z",
  "last_used": null,
  "expires_at": "2026-06-15T00:00:00Z",
  "is_enabled": true,
  "api_key": "mmx_a1b2c3d4e5f6a7b8c3d4e5f6a7b8c3d4"
}

Authentication header formats

API keys can be sent in two ways:

# Header format (preferred)
curl -H "X-API-Key: mmx_a1b2..." https://api.arbitex.ai/api/...

# Authorization header format
curl -H "Authorization: ApiKey mmx_a1b2..." https://api.arbitex.ai/api/...

---

Key storage

Do not store keys in

• Source code or version control (even private repos)
• Configuration files committed to git
• CI/CD pipeline definitions (use secrets instead)
• Log files or debug output
• Chat messages or email

Recommended storage

Platform	Method
Kubernetes	Secret resource (see manifest below)
AWS	Secrets Manager or Parameter Store (SecureString)
Azure	Key Vault
GCP	Secret Manager
CI/CD (GitHub Actions)	Repository or Environment secrets
CI/CD (GitLab)	CI/CD variables (masked, protected)
Local development	.env file in .gitignore

Kubernetes Secret manifest for outpost

apiVersion: v1
kind: Secret
metadata:
  name: arbitex-outpost-keys
  namespace: arbitex
type: Opaque
stringData:
  # Inbound client auth key for /v1/chat/completions
  OUTPOST_API_KEY: "your-strong-random-key-here"
  # HMAC key for audit chain integrity (>= 32 chars)
  AUDIT_HMAC_KEY: "your-32-plus-character-hmac-key-here"
  # Policy bundle HMAC verification key
  POLICY_HMAC_KEY: "shared-secret-matching-platform-signing-key"

Reference the secret in your Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arbitex-outpost
  namespace: arbitex
spec:
  template:
    spec:
      containers:
        - name: outpost
          envFrom:
            - secretRef:
                name: arbitex-outpost-keys

Tip

Use stringData (not data) in Secret manifests to avoid manual base64 encoding. Kubernetes encodes the values automatically.

---

Key rotation

Cloud portal key rotation

Cloud portal keys use soft-delete revocation, which preserves the audit trail. Follow this zero-downtime rotation procedure:

1. Create a new key with a descriptive name indicating the rotation (e.g., "CI Pipeline - rotated 2026-03-15").
2. Update all consumers to use the new key. Both keys are active simultaneously during this window.
3. Verify the new key works by checking last_used_at on the new key.
4. Revoke the old key:

curl -X DELETE https://api.arbitex.ai/v1/orgs/{org_id}/api-keys/{old_key_id} \
  -H "Authorization: Bearer <jwt>"

This sets revoked_at on the old key. The key immediately stops authenticating but the record is preserved.

Platform user key rotation

Platform user keys are hard-deleted on revocation:

1. Create a new key via POST /api/auth/api-keys.
2. Update consumers.
3. Delete the old key:

curl -X DELETE https://api.arbitex.ai/api/auth/api-keys/{old_key_id} \
  -H "Authorization: Bearer <jwt>"

Outpost key rotation

Rotating the outpost inbound key (OUTPOST_API_KEY) requires a coordinated restart:

1. Update the Kubernetes Secret (or environment variable) with the new key value.
2. Update all client applications to send the new Bearer token.
3. Restart the outpost (kubectl rollout restart deployment/arbitex-outpost).
4. During the restart window, clients using the old key will receive 401 Unauthorized.

No grace period

The outpost has a single OUTPOST_API_KEY with no support for multiple simultaneous keys. Plan a brief maintenance window or use a load balancer to drain connections before rotating.

---

Scope and IDOR protection

Cloud portal keys — org-scoped

All cloud API key operations enforce organization isolation at both the route and database query level:

# Route-level: JWT org_id must match URL path org_id
if org_ctx.org_id != org_id:
    raise HTTPException(status_code=403, detail="Forbidden: org_id mismatch")

# DB-level: queries filter by JWT org_id, not URL parameter
select(ApiKey).where(
    ApiKey.id == key_id,
    ApiKey.org_id == org_ctx.org_id  # from JWT, not URL
)

An attacker who obtains a valid JWT for org A cannot use it to list, create, or revoke keys for org B. The org_id in the URL path is verified against the JWT claim, and all database queries use the JWT-sourced org_id.

Platform user keys — user-scoped

Platform key operations filter by the authenticated user’s ID:

# All queries include: WHERE api_keys.user_id = current_user.id
# Attempting to access another user's key returns 404 (not 403 — no oracle)

Admin key management

Platform admins can list and delete any user’s keys via the admin router (/api/v1/admin/api-keys), but admin endpoints require admin-role JWT authentication.

---

Monitoring

Per-key usage stats

Monitor individual key usage via the cloud portal API:

curl https://api.arbitex.ai/v1/orgs/{org_id}/api-keys/{key_id}/usage \
  -H "Authorization: Bearer <jwt>"

Response:

{
  "key_id": "a1b2c3d4-...",
  "requests_24h": 142,
  "requests_7d": 983,
  "requests_30d": 3847
}

This endpoint proxies to the platform’s internal usage tracking service via mTLS. If the platform is unavailable, zero values are returned gracefully.

Indicators of key compromise

Monitor for these patterns that may indicate a compromised key:

Indicator	Description
Unusual request volume	Sudden spike in requests_24h compared to baseline
Geographic anomaly	Requests from unexpected IP ranges (check audit logs)
Off-hours activity	Key usage during periods when the associated service should be idle
Scope escalation attempts	403 errors in audit logs indicating attempts to access other orgs
Multiple concurrent sources	Same key used from different IP addresses simultaneously

last_used_at tracking

Both key systems track the last usage timestamp:

• Cloud keys: last_used_at updated on each authenticated request
• Platform keys: last_used updated with a 60-second debounce (skips DB write if updated within 60s)

Keys with a stale last_used_at value (or null) may be candidates for revocation.

---

Key compromise response

If you suspect an API key has been compromised, follow this procedure immediately:

Step 1 — Revoke the key

Cloud portal key:

curl -X DELETE https://api.arbitex.ai/v1/orgs/{org_id}/api-keys/{key_id} \
  -H "Authorization: Bearer <jwt>"

Platform user key:

curl -X DELETE https://api.arbitex.ai/api/auth/api-keys/{key_id} \
  -H "Authorization: Bearer <jwt>"

Revocation is immediate. The key stops authenticating on the next request.

Step 2 — Review audit logs

Check what actions were taken with the compromised key:

# Search audit logs for the time period the key was active
curl "https://api.arbitex.ai/v1/orgs/{org_id}/audit/events?start_time=2026-03-01T00:00:00Z&limit=500" \
  -H "Authorization: Bearer <jwt>"

Look for:

• Unusual API calls (data exports, configuration changes, key creation)
• Requests from unexpected IP addresses
• Actions outside the key’s intended scope

Step 3 — Rotate related credentials

If the compromised key was used alongside other credentials:

• Outpost OUTPOST_API_KEY: Update the Kubernetes Secret and restart the outpost.
• mTLS certificates: If outpost certs were stored alongside the API key, rotate certs and restart.
• HMAC keys: If AUDIT_HMAC_KEY or POLICY_HMAC_KEY were co-located, rotate and restart.

Step 4 — Create replacement keys

Create new keys with fresh names indicating the rotation:

curl -X POST https://api.arbitex.ai/v1/orgs/{org_id}/api-keys \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "CI Pipeline - rotated after incident 2026-03-15",
    "expires_in_days": 90
  }'

Step 5 — Document and notify

• Record the incident timeline, affected key IDs, and remediation actions taken.
• Notify your security team per your organization’s incident response policy.
• Review access controls for the key storage location that was compromised.

---

Best practices summary

Practice	Rationale
Set expiration on all keys	Limits blast radius of undetected compromise
Use descriptive names	Enables quick identification during incidents
One key per service	Isolates blast radius and simplifies rotation
Monitor last_used_at	Identifies unused keys for revocation
Store in secrets manager	Prevents accidental exposure in code or config
Rotate keys quarterly	Limits exposure window for undetected compromise
Use scopes when available	Applies least-privilege principle
Never log plaintext keys	SHA-256 hash prevents recovery from log compromise
Review audit logs after revocation	Detects unauthorized activity during exposure window

---

Admin operations

Users with the admin role can list and revoke API keys across all user accounts via the /api/v1/admin/api-keys prefix.

List all org keys

Returns metadata for all API keys across all user accounts. Useful for auditing active credentials, identifying stale keys, and enforcing rotation policies. The plaintext key and hash are never included.

curl https://api.arbitex.ai/api/v1/admin/api-keys \
  -H "Authorization: Bearer <admin_session_token>"

Revoke any user’s key

Admins can hard-delete any key regardless of which user owns it. The operation is immediate, permanent, and cannot be recovered.

curl -X DELETE https://api.arbitex.ai/api/v1/admin/api-keys/{key_id} \
  -H "Authorization: Bearer <admin_session_token>"

Returns 204 No Content on success. Use this during incident response to immediately invalidate a compromised credential without requiring access to the key owner’s account.

---

Org-level key governance

Service account pattern

API keys carry the permissions of the user account that created them. There is no mechanism to scope a key to a subset of the creator’s permissions. To achieve least-privilege access across teams:

1. Create a dedicated service account for each team or service (e.g., svc-data-pipeline, svc-ci-prod). Assign that account the minimum required role — user for most integrations, admin only if admin endpoints are required.
2. Generate keys under the service account rather than personal user accounts. This ties the key’s lifecycle to the service account, not to an individual’s employment status.
3. Provision one key per deployment environment (staging, prod) with distinct descriptions.

Example per-team key structure

Service account	Keys	Purpose
svc-pipeline-prod	pipeline-prod-2026-06	Production data pipeline
svc-pipeline-staging	pipeline-staging-2026-06	Staging data pipeline
svc-ci	ci-prod-2026-06	CI/CD integration tests
svc-analytics	analytics-2026-06	Read-only analytics queries

Revoking a service account

When a service account is decommissioned, an admin can revoke all keys owned by that account:

# List all keys across all users to find the service account's keys
curl https://api.arbitex.ai/api/v1/admin/api-keys \
  -H "Authorization: Bearer <admin_token>"

# Revoke each key
curl -X DELETE https://api.arbitex.ai/api/v1/admin/api-keys/{key_id} \
  -H "Authorization: Bearer <admin_token>"

---

Cloud portal scope reference

Cloud portal API keys (arb_/arx_ prefix) support explicit permission scopes. A key may hold one or more scopes.

Scope	Description
dlp:read	Read DLP policies, pattern libraries, and detection results
dlp:write	Create, update, and delete DLP policies and pattern configurations
audit:read	Read audit log entries and export audit data
webhook:write	Create, update, and delete webhook configurations
admin	Full administrative access — includes all above scopes plus user management, SSO configuration, and billing read

Grant the minimum scopes needed. Most CI/CD integrations need only dlp:read + dlp:write. Overly permissive keys increase the blast radius of a credential leak.

---

Key expiry behavior (portal)

Keys with an expiry date automatically stop working at midnight UTC on the configured expiry date.

• The API returns 401 Unauthorized with "reason": "key_expired" for any request made with an expired key.
• Expired keys remain visible in the portal with an Expired status badge — they are retained for audit purposes.
• Expired keys cannot be reactivated — create a new key to restore access.
• The portal displays a warning banner on keys that will expire within 14 days.

{
  "error": {
    "code": "unauthorized",
    "reason": "key_expired",
    "message": "The API key expired on 2026-06-15T00:00:00Z and is no longer valid."
  }
}

Other possible values for the reason field:

Value	Cause
key_missing	No API key header was included in the request
key_malformed	The header value does not match the expected key format
key_revoked	The key has been explicitly revoked
key_expired	The key’s expiry date has passed

---

Expiry monitoring and dormant-key cleanup

Monitoring expiry

The key listing in Settings → API Keys shows the Expires column for each key. An expired key returns 401 Unauthorized on every request with no grace period.

Recommended schedule:

• 30 days before expiry — begin the rotation procedure to allow time to update all consumers.
• At expiry — the key stops working immediately.

Expiry cannot be extended on an existing key. To change the expiry, revoke and recreate.

Monitoring last_used_at

Each key shows a Last Used timestamp updated each time the key successfully authenticates a request. Use it to identify candidates for cleanup:

Condition	Action
Key unused for 30+ days	Candidate for revocation — the service may be decommissioned or the key may be orphaned
Key with no last_used value	Never used — safe to revoke unless the service has not yet launched
Key last used before a known deployment	Verify the service updated to a newer key or investigate whether the old key is still in use

Keys unused for 30+ days are likely orphaned. Revoke them if you cannot identify an owner.

Audit log events

All key lifecycle and authentication events are recorded in the audit log:

Event type	Trigger
api_key.created	A new API key was created
api_key.revoked	An API key was explicitly revoked
api_key.auth_success	An API key was used to successfully authenticate a request
api_key.auth_failed	An authentication attempt using an API key failed

Configure alerts on api_key.auth_failed to detect credential misuse or brute-force attempts.

---

Endpoint reference

Method	Endpoint	Auth	Description
POST	/api/auth/api-keys	Session token	Create a key for the authenticated user
GET	/api/auth/api-keys	Session token	List the authenticated user’s keys
PATCH	/api/auth/api-keys/{key_id}	Session token	Enable or disable a key
DELETE	/api/auth/api-keys/{key_id}	Session token	Revoke a key owned by the authenticated user
GET	/api/v1/admin/api-keys	Admin session token	List all keys across all users
DELETE	/api/v1/admin/api-keys/{key_id}	Admin session token	Revoke any user’s key
POST	/v1/orgs/{org_id}/api-keys	JWT	Create a cloud portal key
GET	/v1/orgs/{org_id}/api-keys	JWT	List cloud portal keys for the org
DELETE	/v1/orgs/{org_id}/api-keys/{key_id}	JWT	Revoke a cloud portal key
GET	/v1/orgs/{org_id}/api-keys/{key_id}/usage	JWT	Per-key usage stats (24h / 7d / 30d)

---

Related documentation

• Security architecture — overall security design
• Outpost architecture — outpost component overview
• Audit Trail & Chain Integrity — audit chain HMAC verification
• Deployment topologies — outpost deployment patterns