Security Hardening

Security Hardening

This guide covers the security hardening controls applied to the Arbitex Platform, including container-level defenses (platform-0258) and application-level features (Epic M / platform-0043).

---

Container Security Controls

Every container in the Arbitex Platform enforces a defense-in-depth set of runtime security controls. These apply to all services: backend, frontend, inference services (NER, DeBERTa, AppGuard, CredInt), PostgreSQL, Redis, PgBouncer, and the OTel collector.

Multi-stage builds

All Dockerfiles use multi-stage builds. Build tools (compilers, package managers, development headers) exist only in the builder stage and are never present in the runtime image. Base images are pinned by SHA-256 digest.

Service	Builder	Runtime
Backend	python:3.12-slim + rust:1.90-slim	Fresh python:3.12-slim
Frontend	node:20-alpine	nginx:1.27-alpine
NER / DeBERTa	CUDA + pip + model bake	Fresh CUDA, pip removed
AppGuard	python:3.12-slim + gcc	Fresh python:3.12-slim, pip removed

Explicit COPY

No Dockerfile uses COPY . .. Every COPY instruction names specific files or directories, preventing test suites, documentation, scripts, and build artifacts from entering the runtime image.

Non-root user (UID 1001)

All Python services run as appuser (UID 1001, GID 1001) with --shell /sbin/nologin and --no-create-home. Application code is owned root:root and world-readable; only explicitly designated writable directories (e.g., /app/uploads) are owned by appuser. The frontend runs as nginx’s built-in UID 101.

Read-only root filesystem

Every service runs with read_only: true in Docker Compose. Writable paths are provided via explicit tmpfs mounts with size limits:

Mount	Size	Services
/tmp	64 MB	All services
/app/.cache	32 MB	Backend (Python cache)
/var/cache/nginx	Default	Frontend
/var/run/postgresql	64 MB	PostgreSQL

PYTHONDONTWRITEBYTECODE=1 is set in all Python images to prevent .pyc write attempts.

Drop ALL Linux capabilities

Every service drops all Linux capabilities via cap_drop: ALL. Only PostgreSQL and Redis add back the minimum required:

• PostgreSQL: CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID
• Redis: SETGID, SETUID

All other services (backend, inference, frontend, OTel) run with zero capabilities.

No privilege escalation

Every service sets:

security_opt:
  - "no-new-privileges:true"
  - "seccomp:runtime/default"

This prevents any process inside the container from gaining additional privileges via setuid binaries, capabilities, or other escalation vectors.

Log rotation

Every service uses the json-file logging driver with rotation:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

Maximum 30 MB of logs per service (3 files × 10 MB). This prevents unbounded disk growth from noisy services.

Additional controls

Control	Description
Python source stripping	All Python services compile to .pyc bytecode and delete .py source files in the runtime image
Network isolation	Two-tier Docker network: backend-net (internal, no external access) for database/Redis; frontend-net for backend-to-frontend
No curl	Health checks use python -c "import urllib.request; ..." or wget — curl is intentionally not installed
Resource limits	Every service has explicit mem_limit and cpus constraints
Read-only volumes	Corpus data (credint-corpus) and model weights are mounted read-only (0444 permissions)
Offline inference	NER and DeBERTa services set TRANSFORMERS_OFFLINE=1 — no outbound network calls to Hugging Face

---

1. Azure Key Vault Secrets Backend

What it does

By default the platform reads sensitive configuration values (database URLs, JWT keys, etc.) from environment variables. The pluggable secrets backend lets you source these values from Azure Key Vault instead, keeping production secrets out of container environment variables and Kubernetes manifests.

Three additional backends are supported alongside the default:

Backend	Description
env (default)	Reads from environment variables — existing behaviour, no change required.
vault / azure_keyvault	Reads from Azure Key Vault. Both values select the same backend.
file	Reads from a JSON file at SECRETS_FILE — useful for local development.

Configuration

# Select the Azure Key Vault backend
SECRETS_BACKEND=vault

# Vault URL (one of the two forms)
AZURE_VAULT_URL=https://myvault.vault.azure.net
# or
AZURE_KEYVAULT_URL=https://myvault.vault.azure.net

Authentication uses DefaultAzureCredential, which automatically picks up:

• Azure Managed Identity (AKS pod identity / workload identity — recommended for production)
• AZURE_CLIENT_ID + AZURE_TENANT_ID + AZURE_CLIENT_SECRET environment variables (service principal)
• Azure CLI credentials (az login — local development)

Secret name translation: Azure Key Vault names cannot contain underscores. The backend automatically translates Python-style names: jwt_secret_key → jwt-secret-key. No manual renaming is required.

Startup behaviour

On startup the platform probes the vault with a connectivity check. If the vault URL is missing or unreachable:

• The application fails fast with a RuntimeError — it will not start.
• This prevents silent startup with empty or default secrets.

Verification

Check the platform startup logs for:

Azure Key Vault backend initialised for https://myvault.vault.azure.net

If the vault is unreachable you will see:

Azure Key Vault at https://myvault.vault.azure.net is unreachable at startup: ...

---

2. CSRF Protection — OAuth and SAML State in Redis

What it does

OAuth 2.0 and SAML SP-initiated login flows both use a random state / RelayState parameter to prevent CSRF attacks. These one-time tokens are now stored in Redis rather than server memory. This provides:

• Correctness in multi-replica deployments — server memory CSRF state breaks when requests are routed to different replicas. Redis provides a shared, consistent store.
• Automatic expiry — tokens expire after 10 minutes via Redis TTL. Expired tokens are cleaned up automatically.
• Single-use enforcement — each token is retrieved and deleted atomically in a Redis pipeline (GET + DEL). Replay attacks are rejected because the key is gone after first use.
• Fail-closed — if Redis is unavailable, the store raises an error and the OAuth/SAML flow is rejected with HTTP 503. The platform will not fall back to a less-secure mechanism.

Configuration

No additional configuration is required beyond setting REDIS_URL. Both OAuth and SAML CSRF stores initialise automatically from the application settings at startup.

REDIS_URL=redis://redis:6379/0

Variable	Required	Description
REDIS_URL	Yes	Redis connection URL. Required for Google OAuth and SAML login flows.

Key format:

• OAuth states: oauth:state:{state_token}
• SAML nonces: saml:state:{nonce}

TTL: 600 seconds (10 minutes) for both OAuth and SAML state tokens.

Verification

After a successful OAuth or SAML login attempt, confirm in Redis that no stale keys remain:

redis-cli keys "oauth:state:*"   # should be empty after login completes
redis-cli keys "saml:state:*"    # should be empty after login completes

If Redis is not configured (REDIS_URL empty), the platform logs:

REDIS_URL is not set — OAuth CSRF state store is disabled. Google OAuth will be unavailable until Redis is configured.

Google OAuth and SAML login will return HTTP 503 until Redis is configured.

---

3. DLP Inference Fail-Closed Mode

What it does

The Platform hosts GPU-accelerated DLP inference microservices (NER and DeBERTa contextual classification). When a microservice is unavailable (crash, OOM, slow startup), the platform must decide whether to allow the request through unscanned or block it.

Fail-closed (closed, default): When the inference microservice is unavailable, the DLP check is treated as a block — the request is denied rather than passed unscanned. This is the safe production default; it prevents sensitive data from reaching AI providers when DLP is degraded.

Fail-open (open): Allows requests through when the inference microservice is unavailable, without DLP scanning. This matches the legacy behaviour and is suitable only for environments where availability takes precedence over data protection.

Configuration

DLP_INFERENCE_FAIL_MODE=closed   # default — block on GPU microservice unavailability
# or
DLP_INFERENCE_FAIL_MODE=open     # legacy — allow unscanned (not recommended for production)

Variable	Default	Values	Description
DLP_INFERENCE_FAIL_MODE	closed	closed, open	Controls MicroserviceNERDetector and DeBERTaValidatorClient behaviour when the inference microservice is unreachable.

Verification

With DLP_INFERENCE_FAIL_MODE=closed, when the NER microservice is stopped, requests that would trigger DLP scanning should receive an error response rather than passing through. Check the platform logs for:

DLP inference unavailable — failing closed (blocking request)

---

4. mTLS Full Chain Verification

What it does

The Platform internal API routes (/v1/internal/) use mTLS to authenticate inbound calls from Hybrid Outposts and other internal services. Previously, verification required specifying a root CA certificate path and optionally an intermediate certificate path as separate files.

Full chain verification (MTLS_CA_BUNDLE) allows a single PEM file containing the complete CA chain (root + optional intermediates concatenated) to be used for trust anchoring. When set:

• The bundle is parsed into a trusted CA pool.
• Leaf certificate chains are verified against the pool.
• MTLS_CA_BUNDLE takes precedence over the legacy CLOUD_CA_CERT_PATH + CLOUD_CA_INTERMEDIATE_PATH configuration.

This simplifies certificate rotation (update one bundle file) and supports complex PKI hierarchies (multi-level intermediate chains).

Configuration

# New — full chain bundle (recommended)
MTLS_CA_BUNDLE=/etc/arbitex/certs/ca-bundle.pem

# Legacy — separate root and optional intermediate (still supported)
CLOUD_CA_CERT_PATH=/etc/arbitex/certs/root-ca.pem
CLOUD_CA_INTERMEDIATE_PATH=/etc/arbitex/certs/intermediate-ca.pem  # optional

Variable	Default	Description
MTLS_CA_BUNDLE	""	Path to a PEM file containing one or more CA certificates (root + intermediates). Takes precedence over CLOUD_CA_CERT_PATH when set.
CLOUD_CA_CERT_PATH	""	Path to root CA PEM (legacy). Used when MTLS_CA_BUNDLE is not set.
CLOUD_CA_INTERMEDIATE_PATH	""	Path to intermediate CA PEM (legacy, optional). Used when MTLS_CA_BUNDLE is not set.

Development note: If neither MTLS_CA_BUNDLE nor CLOUD_CA_CERT_PATH is set, internal endpoints are unavailable (HTTP 503) unless the development bypass is explicitly enabled via internal support configuration. Never disable mTLS for production deployments.

Verification

After setting MTLS_CA_BUNDLE, confirm an outpost heartbeat is accepted:

curl --cert outpost.pem --key outpost.key --cacert ca-bundle.pem \
  https://platform/v1/orgs/{org_id}/outposts/{outpost_id}/heartbeat \
  -H "Content-Type: application/json" -d '{"version":"0.1.0","uptime":60,...}'
# Expect HTTP 200

---

5. SCIM Per-Org Tokens

What it does

SCIM 2.0 provisioning (for syncing users and groups from an IdP like Okta or Azure AD) previously used a global bearer token shared across all organisations. A compromised token could expose provisioning access for every tenant.

Per-org SCIM tokens give each organisation an independently rotatable bearer token stored as a bcrypt hash. Key properties:

• The raw token is returned once at rotation time and cannot be recovered from the database.
• Only the bcrypt hash is persisted (org_scim_tokens table).
• One active token per org at a time — the active token has rotated_at IS NULL.
• Previous tokens are preserved with a rotated_at timestamp for audit history.
• Rotating one org’s token has zero effect on any other org.

CRUD API

Every endpoint below acts on the authenticated admin’s own org. The target org is derived from the credential you authenticate with — no endpoint accepts a client-supplied org identifier as a path segment, a body field, or a header.

Rotate (generate new) token:

POST /api/v1/scim/token/rotate
Authorization: Bearer <admin-user-JWT>

Response (raw token shown once):

{
  "token": "arb_scim_<random>",
  "created_at": "2026-03-12T14:00:00Z"
}

List tokens (history):

POST /api/v1/admin/scim/tokens/list
Authorization: Bearer <admin-user-JWT>

Returns all token records (active + historical) for the caller’s org. Active token has rotated_at: null.

Revoke a single token:

POST /api/v1/admin/scim/tokens/{token_id}
Authorization: Bearer <admin-user-JWT>

Returns 204 No Content. token_id identifies a token belonging to the caller’s own org; it is not an org selector. Revoking every active token suspends SCIM provisioning until a new token is rotated.

Migration from global token

The deployment-wide SCIM_BEARER_TOKEN environment variable has been removed from the platform. It is no longer read, no longer accepted for SCIM authentication, and is ignored if it is still present in your environment. Per-org tokens are the only way to authenticate to the SCIM endpoint.

There is no migration window and no dual-acceptance period: a deployment that still relies on the global token will see SCIM provisioning fail with 401 Unauthorized until a per-org token is issued.

If your environment still carries the variable:

1. In the Admin Portal, navigate to Settings → SCIM for each org.
2. Click Rotate Token to generate a per-org token.
3. Update your IdP configuration (Okta, Entra ID, etc.) with the new token.
4. Repeat for each org.
5. Delete SCIM_BEARER_TOKEN from your platform environment. This is housekeeping only — the platform already ignores it.

Verification

After rotating a token, the SCIM provisioning endpoint should accept requests authenticated with the new token:

curl https://platform/v1/scim/v2/Users \
  -H "Authorization: Bearer arb_scim_<your-token>"
# Expect HTTP 200 with user list

Requests using the old (pre-rotation) token should receive HTTP 401.