Certificate Management

Arbitex uses a three-tier certificate authority hierarchy for mutual TLS (mTLS) between internal components and between the Cloud control plane and Hybrid Outpost deployments. This page describes the CA design, which connections require mTLS, how certificates are managed, and how Outpost certificates are provisioned.

CA hierarchy overview

Root CA
  YubiHSM (offline hardware security module)
  Root private key never leaves hardware
  Signs only the intermediate CA certificate
        │
        ▼
Intermediate CA
  Azure Key Vault (separate HSM-backed key)
  Issues leaf certificates via cert-manager (step-ca issuer)
  Online — used for active certificate signing
        │
        ├── Internal service leaf certificates (service-to-service mTLS)
        ├── Outpost client certificates (issued at Outpost registration)
        └── Cloudflare Origin CA certificate (edge ↔ AKS mTLS)

Root CA

The Root CA is a self-signed certificate stored on a YubiHSM 2 (offline hardware security module). The root private key never leaves the HSM hardware.

After signing the Intermediate CA certificate, the YubiHSM is physically disconnected and taken offline. A root CA compromise would require physical access to the hardware. The offline root ensures that a compromise of the online infrastructure (including the Intermediate CA) cannot forge a new root certificate.

Root CA operations (signing a new intermediate, revocation) require physical access to the YubiHSM and are performed only during planned maintenance events.

Intermediate CA

The Intermediate CA private key is stored in Azure Key Vault (HSM-backed, FIPS 140-2 Level 3). The Intermediate CA is the only CA that issues leaf certificates and is the only CA that performs online operations.

cert-manager uses the step-ca issuer to sign CertificateRequest resources against the Intermediate CA. The Intermediate CA certificate has pathLen=0 — it can only issue leaf certificates, not additional subordinate CAs.

Leaf certificates

All leaf certificates are issued by the Intermediate CA via cert-manager and are valid for a short term (90 days, configurable per issuer). cert-manager handles automatic renewal before expiry.

mTLS connections

Outpost → Cloud control plane

When a Hybrid Outpost registers with the Cloud control plane, it receives a client certificate issued by the Intermediate CA. The Outpost presents this certificate on all subsequent connections to the Cloud API (/outpost/* paths).

The Cloud mTLS middleware (MTLSMiddleware) validates the certificate chain:

Client certificate is present (401 if missing)
Certificate is within its validity period (403 if expired)
Certificate signature chain: leaf → Intermediate CA → Root CA (403 cert_untrusted if chain breaks)
Common Name (CN) is extracted and stored in request.state.mtls_cn for downstream logging

The platform root CA certificate path is configured via CLOUD_CA_CERT_PATH. An optional intermediate CA certificate can be supplied via CLOUD_CA_INTERMEDIATE_PATH for full chain-of-trust verification.

If CLOUD_CA_CERT_PATH is not set (development mode), the middleware logs a warning and blocks internal endpoints with HTTP 503 unless the emergency _SUPPORT_MTLS_BYPASS flag is set. There is no silent fallback to unauthenticated access.

Cloudflare edge → AKS origin

Cloudflare Authenticated Origin Pulls use the Cloudflare Origin CA certificate to verify that traffic reaching the AKS origin server originated from Cloudflare, not arbitrary internet sources. This prevents bypassing Cloudflare WAF/DDoS protection by directly targeting the origin IP.

Internal service-to-service

PostgreSQL, Redis, and Azure Key Vault connections use TLS with Azure-managed certificates. Pod-to-pod communication within the AKS cluster is policy-controlled by Calico (default-deny, explicit allowlist per service pair).

Certificate lifecycle

Issuance

Certificates are issued by cert-manager via the step-ca issuer against the Intermediate CA. cert-manager issues certificates in response to Certificate or CertificateRequest Kubernetes resources.

Certificate type	Validity	Renewal trigger
Internal service leaf	90 days	14 days before expiry
Outpost client certificate	1 year	Outpost re-registration or admin-triggered renewal
Cloudflare Origin CA	1 year	14 days before expiry (cert-manager)

Renewal

cert-manager monitors certificate expiry and automatically submits renewal requests before the expiry threshold. The renewal failure alert triggers if a CertificateRequest remains in Failed state for more than 1 hour — this means a failed renewal generates an alert before the certificate expires.

Outpost certificate renewal

Outpost client certificates are renewed by re-registering the Outpost with the Cloud control plane. The Cloud issues a new certificate during registration. The old certificate remains valid until it expires — there is an overlap window during which both old and new certificates are accepted.

Revocation

Certificate revocation is handled via CRL (Certificate Revocation List) distributed by the Intermediate CA. Immediate revocation (without waiting for certificate expiry) requires:

Adding the certificate serial number to the CRL
Distributing the updated CRL to all services that verify the certificate

For Outpost certificates, revocation is also achieved by deregistering the Outpost from the Cloud control plane — the Cloud API rejects heartbeats and sync requests from deregistered Outposts regardless of certificate validity.

Outpost certificate provisioning

When an Outpost is registered with the Cloud control plane:

The Cloud generates a unique Outpost ID (UUID)
The Intermediate CA issues a leaf certificate with the Outpost ID as the CN (Common Name)
The certificate and associated private key are delivered to the Outpost in the registration response
The Outpost stores the certificate and key on-premises (configurable path)
All subsequent Outpost → Cloud connections present this certificate

The Outpost certificate’s CN (mtls_cn) is logged by the Cloud mTLS middleware on each connection, providing an audit trail of which Outpost instance made each request.

Air-gap deployments

In air-gap deployments, the Outpost cannot reach the Cloud control plane for certificate renewal. Air-gap Outposts require:

Certificate renewal via manual delivery (copy the new certificate file to the Outpost)
A longer initial certificate validity period configured for air-gap issuance
Pre-provisioned certificate bundles included in the air-gap tarball for initial deployment

See Air-gap deployment guide for the certificate provisioning procedure specific to air-gap installations.

Security considerations

CA key compromise scenarios

Scenario	Impact	Mitigation
Intermediate CA key compromised	Attacker can forge leaf certificates	Revoke all issued certs; replace Intermediate CA (requires Root CA)
Root CA key compromised	Attacker can forge a new Intermediate CA	Physical security of YubiHSM; offline storage
Leaf certificate compromised	Attacker can impersonate one service	Revoke the specific leaf; issue replacement

Unsupported key types

The MTLSMiddleware rejects certificates with unsupported public key types (anything other than RSA or EC/ECDSA). If a certificate is rejected with cert_untrusted, verify the certificate uses RSA or EC keys.

mTLS bypass flags

Two emergency bypass flags exist for operator recovery scenarios:

CLOUD_CA_CERT_PATH unset → mTLS enforcement disabled for CLOUD_CA_CERT_PATH paths; 503 returned unless _SUPPORT_MTLS_BYPASS=true
_SUPPORT_MTLS_BYPASS=true → bypasses mTLS validation and logs CRITICAL on every request; for emergency recovery only

These flags must never be active in normal production operation.