Outpost Certificate Management

This guide covers TLS certificate lifecycle management for the Arbitex Hybrid Outpost, including certificate requirements, status monitoring, manual and automatic rotation, backup procedures, and renewal scheduling recommendations.

1. TLS Certificate Requirements

All outpost connections require TLS 1.2 or higher. The outpost will refuse connections from clients that negotiate TLS 1.1 or below.

Format Requirements

Certificates must be PEM-encoded X.509 certificates. The following key types and sizes are supported:

Key Type	Minimum Size	Recommended
RSA	2048-bit	4096-bit
ECDSA	P-256	P-384

Common Name (CN) Matching

The certificate Common Name (CN) must exactly match the hostname at which the outpost is deployed. Wildcard certificates (e.g., *.example.com) are supported when the outpost hostname falls within the wildcard domain. Subject Alternative Names (SANs) are preferred over CN-only certificates for multi-hostname deployments.

Certificate Chain

When using CA-signed certificates, include the full certificate chain in the certificate file. The chain must be ordered leaf-first, root-last:

-----BEGIN CERTIFICATE-----
<leaf certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediate CA certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<root CA certificate>
-----END CERTIFICATE-----

The private key file must contain only the private key for the leaf certificate — do not include the chain in the key file.

Configuration Keys

The outpost reads certificate paths from the following environment variables:

Variable	Default	Description
CERT_PATH	/etc/outpost/tls/cert.pem	Path to PEM-encoded certificate (with chain)
KEY_PATH	/etc/outpost/tls/key.pem	Path to PEM-encoded private key

Both files must be readable by the outpost process. If the outpost starts and cannot read either file, TLS initialization will fail and the process will exit with a non-zero status code.

Caution

Self-signed certificates are supported for development and testing environments but are not recommended for production. In production, use a certificate issued by a trusted CA. Clients connecting to an outpost with a self-signed certificate will receive browser or API client warnings unless the certificate is explicitly trusted.

---

2. Certificate Status Monitoring

The outpost exposes a certificate status endpoint that returns the current state of the loaded TLS certificate. Use this endpoint to integrate certificate expiry monitoring into your existing alerting infrastructure.

Endpoint

GET /admin/api/certs/status
Authorization: Bearer <admin-jwt>

Response Schema

CertStatusResponse

Field	Type	Description
cn	string	Certificate Common Name
issuer	string	Issuer distinguished name
expiry_date	string (ISO 8601)	Certificate expiry timestamp
days_remaining	int	Days until expiry at time of request
needs_rotation	bool	true when days_remaining ≤ 30
serial_number	string	Certificate serial number (hex)
key_type	string	RSA or ECDSA
key_size	int	Key size in bits

Example Response

{
  "cn": "outpost.example.com",
  "issuer": "CN=Example Intermediate CA, O=Example Corp, C=US",
  "expiry_date": "2026-04-15T00:00:00Z",
  "days_remaining": 31,
  "needs_rotation": false,
  "serial_number": "0a:1b:2c:3d:4e:5f:6a:7b",
  "key_type": "ECDSA",
  "key_size": 256
}

When days_remaining drops to 30 or below, the needs_rotation flag transitions to true:

{
  "cn": "outpost.example.com",
  "issuer": "CN=Example Intermediate CA, O=Example Corp, C=US",
  "expiry_date": "2026-04-15T00:00:00Z",
  "days_remaining": 29,
  "needs_rotation": true,
  "serial_number": "0a:1b:2c:3d:4e:5f:6a:7b",
  "key_type": "ECDSA",
  "key_size": 256
}

Monitoring Integration

Poll /admin/api/certs/status on a schedule and trigger an alert when needs_rotation is true. The endpoint is lightweight and safe to call frequently — a 15-minute or hourly polling interval is sufficient for most environments.

Example alert condition:

needs_rotation == true → page on-call: outpost cert expires in ≤30 days
days_remaining <= 7   → page on-call: critical cert expiry imminent

Tip

Integrate certificate status checks into your existing monitoring stack — Prometheus, Datadog, PagerDuty, or similar — by scraping this endpoint on a schedule. The days_remaining field is suitable for use as a Prometheus gauge metric. Export it alongside other outpost health metrics for a unified view of outpost operational status.

---

3. Certificate Rotation Process

The rotation endpoint allows you to replace the active TLS certificate without restarting the outpost. Two modes are supported: self-signed generation and custom certificate upload.

Endpoint

POST /admin/api/certs/rotate
Authorization: Bearer <admin-jwt>
Content-Type: application/json

Mode 1: Self-Signed Certificate Generation

POST with an empty body (or {}). The outpost generates a new self-signed certificate matching the current CN and key type. The new certificate will have a 365-day validity period.

curl -X POST https://outpost.example.com/admin/api/certs/rotate \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{}'

Mode 2: Custom Certificate Upload

POST with a JSON body containing the PEM-encoded certificate and private key. The outpost validates the certificate/key pair, installs it, and performs a graceful reload.

curl -X POST https://outpost.example.com/admin/api/certs/rotate \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "pem_cert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
    "pem_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
  }'

The pem_cert field must contain the full certificate chain (leaf first, root last) if using a CA-signed certificate.

Rotation Response Schema

CertRotationResponse

Field	Type	Description
rotation_success	bool	true if rotation completed successfully
new_cn	string	CN of the newly installed certificate
new_expiry	string (ISO 8601)	Expiry date of the new certificate
previous_expiry	string (ISO 8601)	Expiry date of the replaced certificate
reload_status	string	"completed" or "pending"

Example success response:

{
  "rotation_success": true,
  "new_cn": "outpost.example.com",
  "new_expiry": "2027-03-15T00:00:00Z",
  "previous_expiry": "2026-04-15T00:00:00Z",
  "reload_status": "completed"
}

Graceful TLS Reload

After a successful rotation, the outpost performs a graceful TLS reload:

1. The new certificate is loaded into memory.
2. New TLS connections immediately use the new certificate.
3. Existing TLS connections continue on the old certificate until they complete naturally.
4. No connections are forcibly terminated.

This process results in zero downtime during certificate rotation. The reload_status field in the response reflects whether the reload completed synchronously ("completed") or is still draining existing connections ("pending"). In either case, new connections are already using the new certificate when the API call returns.

Error Responses

HTTP Status	Cause
422 Unprocessable Entity	Certificate and private key do not match
422 Unprocessable Entity	Certificate CN does not match configured hostname
400 Bad Request	Malformed PEM data
401 Unauthorized	Missing or invalid admin JWT

Caution

Always back up the current certificate before initiating a rotation. See Section 5: Backup Before Rotation for the recommended backup procedure. If the rotation fails partway through, having a backup allows you to restore service quickly.

---

4. Auto-Rotation Configuration

The outpost supports automatic certificate rotation for environments where self-signed certificates are acceptable and manual rotation is impractical.

Environment Variable

Variable	Type	Default	Description
CERT_AUTO_ROTATE	bool	false	Enable automatic self-signed certificate rotation

Set CERT_AUTO_ROTATE=true in the outpost environment to enable this feature.

Behavior

When CERT_AUTO_ROTATE is enabled:

• The outpost runs a rotation check every 24 hours.
• If days_remaining ≤ 30, the outpost automatically generates a new self-signed certificate matching the current CN and key type.
• The rotation uses the same graceful reload mechanism as the manual rotation API — no downtime.
• A log entry is written at INFO level when auto-rotation fires: cert_auto_rotate: rotated certificate, new_expiry=<date>.

Limitations

Auto-rotation only generates self-signed certificates. It cannot:

• Request certificates from a CA (ACME/Let’s Encrypt).
• Upload externally issued certificates.
• Renew certificates through an enterprise PKI.

For CA-signed certificate automation, use an external cert management tool (such as cert-manager, Certbot, or your enterprise PKI automation) and trigger rotation via the POST /admin/api/certs/rotate endpoint with the pem_cert/pem_key body.

Recommended Environments

Environment	Recommendation
Development	CERT_AUTO_ROTATE=true — convenient, low operational overhead
Staging	CERT_AUTO_ROTATE=true — mirrors dev, no CA dependency
Production	CERT_AUTO_ROTATE=false — use CA-signed certs with external automation

Tip

For production environments using Let’s Encrypt or cert-manager, configure your ACME client to write the renewed certificate to CERT_PATH and KEY_PATH, then call POST /admin/api/certs/rotate with the new PEM data. This gives you CA-signed certificates with automated renewal and zero-downtime rotation.

---

5. Backup Before Rotation

Back up the active certificate and private key before every rotation — whether manual or scripted. A certificate backup takes seconds and eliminates recovery risk if a rotation goes wrong.

API-Level Backup

The standard config backup endpoint includes certificate files:

curl -X POST https://outpost.example.com/admin/api/config/backup \
  -H "Authorization: Bearer $ADMIN_JWT" \
  --output outpost-backup-$(date +%Y%m%d).tar.gz

The backup archive includes cert.pem and key.pem from the configured CERT_PATH and KEY_PATH locations.

Manual File Backup

For direct filesystem access, copy the certificate files to a date-stamped backup directory before rotation:

#!/usr/bin/env bash
# backup-certs.sh — run before every certificate rotation

set -euo pipefail

CERT_PATH="${CERT_PATH:-/etc/outpost/tls/cert.pem}"
KEY_PATH="${KEY_PATH:-/etc/outpost/tls/key.pem}"
BACKUP_DIR="/etc/outpost/tls/backups/$(date +%Y%m%d-%H%M%S)"

mkdir -p "$BACKUP_DIR"
cp "$CERT_PATH" "$BACKUP_DIR/cert.pem"
cp "$KEY_PATH"  "$BACKUP_DIR/key.pem"

echo "Certificate backed up to: $BACKUP_DIR"
echo "Cert fingerprint: $(openssl x509 -in "$BACKUP_DIR/cert.pem" -noout -fingerprint -sha256)"

Restore Procedure

To restore a backed-up certificate after a failed rotation:

# Replace active cert with backup
cp /etc/outpost/tls/backups/<timestamp>/cert.pem "$CERT_PATH"
cp /etc/outpost/tls/backups/<timestamp>/key.pem  "$KEY_PATH"

# Trigger reload via rotation API using the backed-up PEM data
curl -X POST https://outpost.example.com/admin/api/certs/rotate \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d "{
    \"pem_cert\": \"$(awk '{printf "%s\\n", $0}' "$CERT_PATH")\",
    \"pem_key\":  \"$(awk '{printf "%s\\n", $0}' "$KEY_PATH")\"
  }"

---

6. Renewal Schedule Recommendations

Rotation Threshold

Rotate at 30 days remaining — this matches the needs_rotation threshold returned by /admin/api/certs/status. Starting at 30 days gives you a comfortable window to complete renewal, handle any CA processing delays, and verify the new certificate before the old one expires.

CA-Signed Certificate Renewal

For certificates issued by an internal or external CA:

1. Initiate CSR generation at 30 days remaining. Submit to your CA.
2. Receive and validate the new certificate. Verify the CN, SANs, and expiry date.
3. Back up the current certificate (see Section 5).
4. Upload via rotation API:

   curl -X POST https://outpost.example.com/admin/api/certs/rotate \
     -H "Authorization: Bearer $ADMIN_JWT" \
     -H "Content-Type: application/json" \
     -d '{
       "pem_cert": "<new-cert-chain-pem>",
       "pem_key":  "<new-private-key-pem>"
     }'

5. Verify the new certificate:

   curl -s https://outpost.example.com/admin/api/certs/status \
     -H "Authorization: Bearer $ADMIN_JWT" | jq .

   Confirm days_remaining reflects the new expiry and needs_rotation is false.

Self-Signed (Dev / Staging)

Enable CERT_AUTO_ROTATE=true. The outpost handles renewal automatically. No manual action required.

Monitoring Checklist

Check	Frequency	Action
GET /admin/api/certs/status	Daily	Alert if needs_rotation is true
Certificate expiry trend	Weekly	Review days_remaining — confirm it is decreasing as expected
Rotation test	Monthly	Test rotation procedure in staging before next production rotation is due
Backup verification	Monthly	Verify cert backups are complete and restorable

---

7. Troubleshooting

Common Issues

Symptom	Cause	Resolution
422 Unprocessable Entity on rotation	Certificate and private key do not match	Ensure pem_cert and pem_key are from the same key pair
TLS handshake failure on client connections	Expired certificate	Rotate immediately; use self-signed generation if CA-signed cert is unavailable
Browser certificate warning / CN mismatch error	Certificate CN does not match the outpost hostname	Issue a new certificate with the correct CN or add a SAN for the hostname
Outpost fails to start: cannot read cert file	File permission error on CERT_PATH or KEY_PATH	Ensure the outpost process user has read access to both files: chmod 640 cert.pem key.pem && chown outpost:outpost cert.pem key.pem
needs_rotation: true immediately after rotation	Rotation installed a certificate already within the 30-day window	Verify the new certificate expiry — the uploaded cert may itself be near expiry
reload_status: pending remains indefinitely	Long-lived connections not draining	Check for stuck WebSocket or keep-alive connections; restart the outpost if drain does not complete within 10 minutes

Certificate Verification Commands

Inspect the loaded certificate:

openssl x509 -in /etc/outpost/tls/cert.pem -noout -text

Check expiry only:

openssl x509 -in /etc/outpost/tls/cert.pem -noout -enddate

Verify the certificate chain:

openssl verify -CAfile /etc/outpost/tls/cert.pem /etc/outpost/tls/cert.pem

Verify the certificate against an explicit CA bundle:

openssl verify -CAfile /path/to/ca-bundle.pem /etc/outpost/tls/cert.pem

Confirm that the certificate and private key match (the modulus hashes should be identical):

openssl x509 -noout -modulus -in /etc/outpost/tls/cert.pem | openssl md5
openssl rsa  -noout -modulus -in /etc/outpost/tls/key.pem  | openssl md5

For ECDSA keys:

openssl x509 -noout -pubkey -in /etc/outpost/tls/cert.pem | openssl md5
openssl ec   -pubout        -in /etc/outpost/tls/key.pem  | openssl md5

Tip

Use openssl s_client -connect hostname:port to verify the certificate actually presented by the outpost over the network — this confirms the running process has loaded the expected certificate, not just that the file on disk is correct:

openssl s_client -connect outpost.example.com:443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate

---

Related Documentation

• Outpost Security Hardening — TLS baseline configuration, cipher suite policy, and mutual TLS (mTLS) setup
• API Reference — Batch 34 — Full endpoint specifications for /admin/api/certs/status and /admin/api/certs/rotate
• Outpost Administration — budget enforcement, CredInt, health monitoring, JWT validation, PVC recovery, security hardening, SIEM direct integration