IR-9: Certificate Compromise

A certificate compromise means a private key in the Arbitex PKI hierarchy has been leaked, an unauthorized certificate was issued, or a CA itself has been breached. The response depends heavily on which tier is affected. The platform uses a three-tier CA hierarchy: Root CA (YubiHSM 2, offline), the issuing CA — today CN=Arbitex Production CA, held as an EC-HSM P-384 key in Azure Key Vault — and leaf certificates on a 90-day lifetime. Outpost mTLS certificates are issued at registration and identify each outpost by CN.

Two naming points matter when you are working under time pressure:

• The middle tier is called issuing, not “intermediate.” The deployed hierarchy has exactly two CA tiers — {"tiers": ["root", "issuing"]} in the PKI store’s hierarchy.json, and two certificates in ca-bundle.pem. arbitex-pki still accepts --tier intermediate because the tier is defined in code, but nothing in this deployment occupies it, so passing it will not find your CA.
• Leaf issuance runs through the arbitex-pki CLI, not cert-manager. See Leaf issuance is not cert-manager before running any kubectl step in this runbook.

For the PKI hierarchy in detail, see Certificate Management. For mTLS topology, see TLS Topology.

Leaf issuance is not cert-manager

Read this before running any kubectl command below. Several steps in this runbook are written against cert-manager Certificate / CertificateRequest resources. cert-manager is not installed on the production clusters, so those commands return the server doesn't have a resource type "certificate" rather than doing anything. They are retained — not deleted — because the shape of each step is still the right shape, but you must translate them to the real issuance path before acting.

What is actually true, and what measured it:

Claim	Evidence
cert-manager is absent from the prod admin cluster	kubectl --context=abtx-prod-use2-admin-aks get clusterissuer → the server doesn't have a resource type "clusterissuer"; get pods -A | grep -i cert → no output. Measured 2026-08-02 by tooling-0057, recorded in arbitex-admin deploy/helm/arbitex-admin/values.yaml
cert-manager is absent from the demo cluster	arbitex-coordinator _todo/admin-demo-colocated-standup-2026-07-23.md
cert-manager as the internal leaf issuer is an open FR, not shipped	arbitex-coordinator _todo/fr-akv-intermediate-ca-standup-and-key-shred.md — “Steps 5 (AKV import) + 6 (cert-manager) are the unfinished pieces”
The 90-day leaf lifetime is real, and comes from the platform PKI	arbitex-platform backend/app/pki/ca.py:145 LEAF_LIFETIME_DAYS = 90, :151 LEAF_RENEW_BEFORE_DAYS = 30
Leaf issuance and renewal run through arbitex-pki	arbitex-platform backend/app/pki/ + scripts/arbitex-pki.py (issue-server, issue-du, issue-admin-workload, renew, revoke, crl)
The admin plane’s client cert is renewed by a CronJob, not a CRD	arbitex-admin deploy/helm/arbitex-admin/templates/cronjob-cert-renewal.yaml + files/renew_client_cert.py, which patch a Kubernetes Secret

Rewriting each cert-manager procedure below against arbitex-pki is tracked as follow-up work; the commands in this runbook have not yet been re-authored or re-tested against it.

---

1. Detection

Certificate compromise surfaces through PKI management anomalies or unexpected certificate presentations:

Signal	Source	Notes
Unexpected CN on mTLS connection	Platform request.state.mtls_cn in access logs	May indicate forged outpost cert
Unauthorized CertificateRequest in Kubernetes	kubectl get cr -A	cert-manager resource created outside normal flow
Azure Key Vault audit log — unauthorized key operation	Key Vault diagnostics	Private key read or sign operation by unexpected principal
Outpost presenting cert with wrong serial number	Platform mTLS validation log	Cert replaced without re-enrollment
Customer report — certificate validation failure	Support ticket or out-of-band	Unexpected cert presented to customer’s trust store
cert-manager alerts on issuance failure or renewal anomaly	Prometheus / Alertmanager	Only fires where cert-manager runs — not the production clusters
Failed cert-renewal CronJob in the admin namespace	Kubernetes Job status / Alertmanager	The renewal job is designed so a failed Job is the alarm
Unexpected certificate in trust store update	PKI management tooling	Unauthorized CA cert insertion

The two cert-manager rows above are retained for environments that run it; on the production clusters they are silent by construction. Do not read their silence as an all-clear.

Check current certificate state for a specific service or outpost:

# Check the certificate currently presented on the API endpoint
openssl s_client -connect api.arbitex.ai:443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -text | grep -E "(Subject|Issuer|Not After|Serial)"

# Review what the PKI actually issued — the authoritative ledger
arbitex-pki list --store "$ARBITEX_ROOT/secrets/pki"

# cert-manager objects, where cert-manager is installed (not the production clusters)
kubectl get certificaterequest -A -o wide
kubectl get certificate -n arbitex -o wide

Query Azure Key Vault audit logs for unauthorized key operations:

az monitor activity-log list \
  --resource-group <rg> \
  --resource-type "Microsoft.KeyVault/vaults" \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query "[?contains(operationName.value, 'sign') || contains(operationName.value, 'unwrapKey')]" \
  --output table

---

2. Severity classification

The severity is determined by which tier of the PKI hierarchy is affected:

Tier	Compromise condition	Severity
Leaf cert (single service)	Private key leaked or cert forged	HIGH
Leaf cert (outpost)	Outpost cert leaked — one customer’s mTLS at risk	HIGH
Issuing CA (Arbitex Production CA)	Private key accessible outside Key Vault HSM	CRITICAL
Root CA	YubiHSM access compromised or key exported	CRITICAL
Multiple outpost certs	Bulk outpost cert compromise	CRITICAL

Escalate to CRITICAL immediately if:

• Any indication the issuing CA private key was accessible outside the Azure Key Vault HSM. The key is stored kty=EC-HSM with export withheld, so a successful export is itself the incident
• Root CA YubiHSM was physically accessed by an unauthorized person
• Multiple outpost CNs are presenting unexpected certificates simultaneously

---

3. Immediate actions (first 15 minutes)

The correct immediate actions depend on which tier is affected. Identify the compromised tier before proceeding.

Step	Action	Command
1	Declare incident	POST /api/staff/incident/declare
2	Freeze audit log	POST /api/staff/emergency/audit/freeze
3	Identify compromised tier	Certificate inspection + Key Vault logs
4A (leaf)	Revoke the specific cert	arbitex-pki revoke — see Section 5
4B (issuing/Root)	Disconnect all outposts	POST /api/staff/emergency/outpost/disconnect for each
5	Issue replacement cert or new CA	Tier-specific procedure — see Sections 5 and 6

Step 1 — Declare incident (use severity matching the affected tier):

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/declare" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Certificate compromise — <leaf / issuing CA / Root CA> affected",
    "severity": "HIGH",
    "playbook": "IR-9",
    "notes": "Affected cert: <CN or CA name>. Tier: <leaf/issuing/root>."
  }' | jq '{incident_id: .id, ic_expires: .ic_window_expires}'

Step 2 — Freeze audit log:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/audit/freeze" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-9: Certificate compromise — preserving evidence"
  }'

Step 4B — Disconnect all outposts (issuing/Root CA compromise only):

When the issuing or Root CA is compromised, all leaf certs signed by it are suspect. Disconnect all outposts immediately to prevent use of potentially-compromised mTLS credentials:

# Disconnect each outpost — run for every registered outpost
for OUTPOST_ID in $(curl -s "https://api.arbitex.ai/api/v1/admin/outposts" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq -r '.outposts[].id'); do
  echo "Disconnecting outpost $OUTPOST_ID"
  curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/outpost/disconnect" \
    -H "Authorization: Bearer $STAFF_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"outpost_id\": \"${OUTPOST_ID}\", \"incident_id\": \"<incident_id>\"}"
done

---

4. Investigation

Determine the exact scope of the compromise before executing recovery. The investigation procedure varies by tier.

Leaf certificate investigation

The two kubectl blocks in this subsection assume cert-manager; see Leaf issuance is not cert-manager. The issuance ledger arbitex-pki list is the authoritative record of what was issued and when.

# Check the cert's serial number against the cert-manager record
kubectl get certificate <cert-name> -n arbitex -o jsonpath='{.status.notAfter}'
kubectl get secret <cert-secret> -n arbitex -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -text | grep -E "(Serial|Subject|Issuer)"

# Check for duplicate certificate presentations in the access log
grep "mtls_cn=<expected_cn>" /var/log/arbitex/access.log | \
  awk '{print $1, $NF}' | sort | uniq -c

If two different serial numbers are presented for the same CN, an unauthorized duplicate was issued.

Check cert-manager issuance history

# Review cert-manager events for unexpected issuance
kubectl get events -n arbitex \
  --field-selector reason=Issued \
  --sort-by='.metadata.creationTimestamp' | tail -50

# Check for CertificateRequest objects created outside normal deployment
kubectl get certificaterequest -A \
  -o custom-columns='NAME:.metadata.name,ISSUER:.spec.issuerRef.name,CREATED:.metadata.creationTimestamp' \
  | sort -k3

Issuing CA investigation

Key Vault holds a key, not a certificate. The vault stores exactly one CA object — the key arbitex-production-ca (kty=EC-HSM, crv=P-384, keyOps=[sign,verify], export withheld) — and zero certificates. So there is no stored certificate to compare a serial or thumbprint against, and az keyvault certificate show returns not-found whatever name you give it. Inspect the key, and prove the pairing with arbitex-pki:

# Inspect the issuing CA KEY (not a certificate — the vault holds no certificates)
az keyvault key show \
  --vault-name <vault_name> \
  --name arbitex-production-ca \
  --query "{kid:key.kid, kty:key.kty, crv:key.crv, enabled:attributes.enabled}"

An unexpected key version in kid, or kty no longer reading EC-HSM, means the key was replaced or re-imported — treat either as confirmation of compromise.

The authoritative check that the vault key still matches the CA certificate in the PKI store is:

# Proves the issuing tier's signing key matches its certificate (cert pubkey == signer pubkey)
arbitex-pki verify-custody --store "$ARBITEX_ROOT/secrets/pki" --tier issuing

# Show where each tier's key lives, plus CRL freshness
arbitex-pki custody --store "$ARBITEX_ROOT/secrets/pki"

A MATCH result means the vault still holds the key that belongs to CN=Arbitex Production CA. A mismatch means the certificate and the signing key have diverged — an unauthorized re-issuance or key replacement.

# Check Key Vault access log for sign/unwrapKey operations in the last 7 days
az monitor diagnostic-settings list \
  --resource <keyvault_resource_id>

Review Key Vault access policies for any recently added or unexpected principals:

az keyvault show --vault-name <vault_name> \
  --query "properties.accessPolicies[*].{objectId:objectId, permissions:permissions.keys}"

Verify the outpost CN inventory

# List all registered outpost CNs
curl -s "https://api.arbitex.ai/api/v1/admin/outposts" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '[.outposts[] | {id, cn: .mtls_cn, org_id, last_connected}]'

# Check access logs for any CNs not in the registered inventory
grep "mtls_cn=" /var/log/arbitex/access.log | \
  grep -oP 'mtls_cn=\K[^,\s]+' | sort | uniq > observed-cns.txt

Any CN appearing in access logs that is not in the registered outpost inventory is a forged certificate.

There is no mTLS bypass to check for

Earlier revisions of this runbook told you to grep for activation of a SUPPORT_MTLS_BYPASS env var. Do not spend incident time on that. The flag was retired in platform-3355 (field, env binding and middleware branch all removed), and the name-only startup tripwire that briefly outlived it was removed in platform-3358 by PO ruling. Setting the name today produces no bypass and no refusal and no log line — it is inert, so its absence from the logs proves nothing either way.

The observable substitute is the CN inventory comparison directly above: mTLS is enforced unconditionally, so every accepted connection carries a real client certificate, and any CN in the access logs that is absent from the registered outpost inventory is a forged certificate.

---

5. Containment

Leaf certificate containment

Revoke the compromised cert and trigger immediate re-issuance. The real revocation path is arbitex-pki revoke, which revokes the serial and republishes the CRL in one step:

# Revoke by serial and republish the CRL
arbitex-pki revoke --store "$ARBITEX_ROOT/secrets/pki" \
  --serial <serial> --reason keyCompromise

# Find the serial in the issuance ledger if you do not have it
arbitex-pki list --store "$ARBITEX_ROOT/secrets/pki"

The kubectl block below is written against cert-manager, which is not installed on the production clusters — see Leaf issuance is not cert-manager. It is retained for the shape of the step and for any environment where cert-manager is present:

# Delete the certificate object — cert-manager will re-issue
kubectl delete certificate <cert-name> -n arbitex

# Or: annotate to force renewal without deletion
kubectl annotate certificate <cert-name> -n arbitex \
  cert-manager.io/issuing-reason="IR-9: Private key compromise" --overwrite

# Verify re-issuance
kubectl get certificate <cert-name> -n arbitex -w

For outpost leaf cert compromise, revoke the outpost’s registration and re-issue:

# Remove the outpost registration (forces re-enrollment with a new cert)
curl -s -X DELETE "https://api.arbitex.ai/api/v1/admin/outposts/<outpost_id>" \
  -H "Authorization: Bearer $STAFF_TOKEN"

The customer must then re-run the outpost registration flow to receive a new mTLS certificate.

Issuing CA containment

When the issuing CA is compromised, all certs signed by it are invalid. This is a catastrophic scenario requiring out-of-band operations:

1. All outposts are already disconnected (Step 4B above).
2. Remove the compromised issuing CA from all service trust bundles.
3. Issue a new issuing CA from the Root CA — this requires physical access to the YubiHSM.
4. Re-issue all leaf certs signed by the new issuing CA.
5. Re-enroll all outposts with certificates signed by the new issuing CA.

The PKI store keeps two files that matter here: ca.crt, the root anchor referenced by CLOUD_CA_CERT_PATH, and ca-bundle.pem, the assembled chain nginx serves as ssl_client_certificate. Verify against those, not against an arbitex-intermediate-ca.pem — no such file exists:

# Verify a leaf still chains to the root once the compromised issuing CA is out of the bundle
openssl verify -CAfile "$ARBITEX_ROOT/secrets/pki/ca.crt" \
  -untrusted "$ARBITEX_ROOT/secrets/pki/ca-bundle.pem" \
  /etc/ssl/certs/arbitex-api.pem

No emergency mTLS bypass exists

There is no way to bypass mTLS, and this is deliberate. If you are locked out, the only path back is certificate re-issuance — Section 6 below.

Earlier revisions of this runbook instructed you to set a SUPPORT_MTLS_BYPASS Key Vault secret as a last resort. That instruction has been removed because it does not work and never announces that it does not work:

• The flag was fully retired in platform-3355 — the settings field, its env binding, and the middleware branch that honoured it were all deleted. There is no workstation or maintenance carve-out.
• The name-only startup tripwire that was briefly kept afterwards was itself removed in platform-3358 by explicit PO ruling.

The consequence is the dangerous one: setting that variable now raises no error, logs nothing, and grants nothing. An operator who sets it and restarts sees a clean startup and may reasonably conclude the bypass is active while access stays broken. Do not attempt it.

If certificate re-issuance cannot restore access, escalate to the incident commander for an out-of-band recovery decision rather than looking for a bypass.

---

6. Recovery

Leaf cert recovery

After the new cert is issued, verify the new cert is being presented. Re-issuance generates a new key — arbitex-pki renew requires a freshly generated CSR and does not carry the old key forward — so both tls.crt and tls.key change together, and a deploy that updates only one fails the handshake:

# Verify new serial number from the Kubernetes Secret the renewal patched
kubectl get secret <cert-secret> -n arbitex \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -serial

# The cert-manager readiness check below applies only where cert-manager is installed
# (not the production clusters) — see "Leaf issuance is not cert-manager" above
kubectl get certificate <cert-name> -n arbitex \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

# Test mTLS connectivity
openssl s_client -connect api.arbitex.ai:443 \
  -cert /path/to/client.crt -key /path/to/client.key \
  -CAfile /path/to/ca-bundle.pem </dev/null 2>&1 | grep "Verify return code"

For outpost recovery, work with the customer to re-run outpost enrollment:

# Check outpost re-enrollment status after customer runs registration
curl -s "https://api.arbitex.ai/api/v1/admin/outposts/<new_outpost_id>" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq '{id, cn: .mtls_cn, status}'

Issuing CA recovery (condensed)

Step	Action
1	Bring YubiHSM online in a secured physical environment
2	Generate new issuing CA key pair in Key Vault and sign with Root CA
3	Import the new issuing CA key to Azure Key Vault; the certificate goes in the PKI store, not the vault (the vault holds no certificates)
4	Reassemble and distribute ca-bundle.pem to all platform services
5	Restart all platform instances to pick up new trust bundle
6	Re-issue all leaf certificates via arbitex-pki
7	Re-enroll all outposts against the new CA
8	Republish the CRLs (arbitex-pki crl) so ssl_crl does not refuse valid leaves

Confirm the new tier before declaring recovery:

arbitex-pki verify-custody --store "$ARBITEX_ROOT/secrets/pki" --tier issuing

Verify platform health after cert recovery:

curl -s "https://api.arbitex.ai/readyz" | jq '.'
curl -s "https://api.arbitex.ai/health/deep" | jq '{status, db, redis, providers}'
curl -s "https://api.arbitex.ai/healthz" | jq '.'

---

7. Communication

Internal notification (immediately on CRITICAL — issuing or Root CA)

Subject: [CRITICAL — IR-9] Certificate CA compromise — all outposts disconnected

Incident ID: [INCIDENT_ID]
Tier: [Issuing CA (Arbitex Production CA) / Root CA]
Detected: [ISO_TIMESTAMP]

ALL outposts have been disconnected. All customers with Hybrid Outpost deployments are currently offline.

Physical access to the YubiHSM is required to issue a new issuing CA.
Coordinate with [PHYSICAL_ACCESS_CONTACT] immediately.

Impact: [N] customer outposts offline. Platform API (cloud-only customers) unaffected.
ETA for re-enrollment: [ESTIMATE]

IC: [IC_NAME] — [CONTACT]

Customer notification — leaf cert affecting one outpost

Send to the customer’s technical contact:

Subject: [Security Notice] Outpost re-enrollment required for your Arbitex Hybrid Outpost

We have revoked and re-issued the mTLS certificate for your Arbitex Hybrid Outpost as part of a security response.

Action required:
Your outpost administrator must re-run the outpost registration command to receive a new certificate and restore connectivity.

Registration command:
  ./arbitex-outpost enroll --token <new_enrollment_token>

Your outpost will be offline until re-enrollment is complete.

Incident reference: [INCIDENT_ID]
Contact: [email protected] | [DIRECT_PHONE]
Deadline: Please complete re-enrollment within [TIMEFRAME].

Customer notification — issuing/Root CA compromise (all outpost customers)

Subject: [Security Notice] Required: Arbitex Hybrid Outpost re-enrollment

We are contacting you regarding a security event affecting the Arbitex PKI certificate hierarchy.

As a precautionary measure, we have:
- Disconnected all customer outposts from the platform
- Revoked all existing outpost certificates
- Issued a new certificate authority

Action required — your outpost administrator must re-enroll your outpost to restore connectivity:

  ./arbitex-outpost enroll --token <new_enrollment_token>

A new enrollment token specific to your organization will be sent in a separate communication within [TIMEFRAME].

We apologize for the disruption. This action was taken to protect your security.

Incident reference: [INCIDENT_ID]
Contact: [email protected] | [DIRECT_PHONE]

---

8. Post-incident

Evidence preservation checklist:

Item	Status
Compromised certificate serial numbers recorded	[ ]
Key Vault audit logs exported for unauthorized key operations	[ ]
cert-manager event log captured	[ ]
All observed mTLS CNs compared against registered inventory	[ ]
arbitex-pki verify-custody --tier issuing returns MATCH on the new CA	[ ]
New CA thumbprints and serial numbers recorded	[ ]
All customer re-enrollment confirmations collected	[ ]
mTLS verification test results for all services recorded	[ ]

Close the incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/close" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Compromised cert(s) revoked, replacements issued, affected outposts re-enrolled, mTLS verified.",
    "root_cause": "<private_key_leaked / unauthorized_issuance / key_vault_access / physical_access>",
    "data_exfiltrated": false
  }'

Lessons learned template:

• Was the compromised private key stored outside the HSM at any point?
• Were Key Vault access policies reviewed recently? Were any stale service principals removed?
• How was the unexpected certificate presentation detected — active monitoring or reactive report?
• Is the outpost CN inventory automated and continuously verified against access logs?
• Should 90-day leaf cert rotation be shortened for higher-sensitivity services?
• Did any step in this runbook fail to execute as written? Stale commands in an IR runbook cost minutes that the incident does not have.

Related playbooks:

• If unauthorized access to Azure Key Vault is confirmed: follow IR-5: Database Breach Suspected for infrastructure-level access
• If an outpost was used for unauthorized data access using a forged cert: follow IR-4: Outpost Compromise
• If the compromise occurred via a compromised staff credential: follow IR-12: Insider Threat