Audit Trail & Chain Integrity

Audit Trail & Chain Integrity

Arbitex implements a two-tier audit system that records every AI gateway interaction across both the cloud platform and on-premises outpost deployments. Every audit entry is cryptographically chained using HMAC-SHA256, producing a tamper-evident log that proves no events have been inserted, deleted, or modified after recording.

This document covers the architecture of both tiers, the HMAC chain integrity algorithm, chain verification, compliance use cases, and OCSF mapping.

For the audit log schema field reference, see Audit Data Model Reference. For admin API endpoints, see Audit log export.

---

Architecture overview

┌─────────────────────────────────────────────────────────────────┐
│                      Cloud Platform                             │
│                                                                 │
│  ┌──────────────┐    ┌───────────────┐    ┌──────────────────┐  │
│  │  API Gateway  │───▶│  SinkManager  │───▶│  audit_logs (PG) │  │
│  └──────────────┘    │  + HMACChain  │    └──────────────────┘  │
│                      │               │    ┌──────────────────┐  │
│                      │               │───▶│  audit.jsonl      │  │
│                      │               │    └──────────────────┘  │
│                      │               │    ┌──────────────────┐  │
│                      │               │───▶│  webhook / SIEM   │  │
│                      └───────────────┘    └──────────────────┘  │
│                             ▲                                   │
│                             │ mTLS sync                         │
│                             │                                   │
├─────────────────────────────┼───────────────────────────────────┤
│                      Outpost                                    │
│                             │                                   │
│  ┌──────────────┐    ┌──────┴────────┐    ┌──────────────────┐  │
│  │  DLP Scanner  │───▶│ AuditLogger   │───▶│  audit.jsonl     │  │
│  └──────────────┘    │ + HMAC chain  │    └──────────────────┘  │
│                      └───────────────┘    ┌──────────────────┐  │
│                             │             │  Ring buffer (50) │  │
│  ┌──────────────┐           │             │  (DLP scan events)│  │
│  │  Admin API    │◀──────────┘            └──────────────────┘  │
│  │  :8301        │                        ┌──────────────────┐  │
│  └──────────────┘                         │  SQLite queue     │  │
│                                           │  (degradation)    │  │
│                                           └──────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Tier 1 — Platform audit events are stored durably in PostgreSQL (audit_logs table). They cover all API interactions, user actions, policy evaluations, and inference requests processed through the cloud platform. Events are HMAC-chained and can be fanned out to multiple sinks (database, JSONL file, webhook, Splunk HEC).

Tier 2 — Outpost audit events are stored in a local JSONL file (audit.jsonl) with an in-memory DLP scan ring buffer (last 50 events) for quick admin access. Outpost events are independently HMAC-chained and periodically synced to the platform via mTLS.

---

Platform audit events

Event creation flow

When the platform processes a request, the audit pipeline executes in a fixed order:

1. IP attachment — src_ip and dst_ip are injected into the event dict before HMAC computation. Empty IPs are coerced to empty strings ("") rather than None to ensure deterministic JSON serialization.

2. HMAC chaining — SinkManager.dispatch() calls HMACChain.chain_event(event), which injects hmac, previous_hmac, and hmac_key_id into the event.

3. Sink fan-out — the signed event is dispatched to all configured sinks (AUDIT_SINKS env var, default "db"). Valid sinks: db, jsonl, webhook, splunk_hec.

4. GeoIP enrichment — country, region, city, ISP, and ASN fields are derived from src_ip and dst_ip after HMAC computation. These enrichment fields are intentionally excluded from the HMAC chain because they are derived from GeoIP database versions that change over time (MaxMind updates monthly, ARIN reassigns netblocks quarterly).

Enrichment field immutability

GeoIP enrichment is a one-time operation at event creation. Enrichment fields are never backfilled or updated on existing audit rows. This design ensures that a GeoIP database update does not retroactively alter historical audit records, which would complicate compliance audits.

AuditLog model structure

The complete field reference is in Audit Data Model Reference. The fields relevant to chain integrity are:

Field	Type	Description
id	UUID	Primary key, generated via uuid4().
hmac	string(64)	HMAC-SHA256 hex digest for this event. null when chaining is disabled.
previous_hmac	string(64)	Digest of the preceding event. GENESIS_HMAC for the first event.
hmac_key_id	string(64)	Key version identifier. Default "default". Supports key rotation.

HMAC fields included in computation

The following fields are serialized via json.dumps(content, sort_keys=True, default=str) and included in the HMAC digest. Fields not in this list (GeoIP enrichment, CredInt analysis, outpost provenance) are excluded:

action, conversation_id, cost_estimate, dst_ip, ip_address,
latency_ms, metadata, model_id, prompt_text, provider,
response_text, src_ip, tenant_id, timestamp,
token_count_input, token_count_output, user_id

Note

src_ip and dst_ip were added to the HMAC payload in v2 (migration 043). The hmac_key_id field distinguishes v1 and v2 computations during chain verification.

---

Platform HMAC computation

Algorithm

The platform uses HMAC-SHA256 with a configurable key and key ID:

HMAC_KEY       = AUDIT_HMAC_KEY (env var, minimum recommended: 32 chars)
HMAC_KEY_ID    = AUDIT_HMAC_KEY_ID (env var, default: "default")
GENESIS_HMAC   = "0" × 64  (64 zero characters)

For each event:
  content     = event dict with [hmac, previous_hmac, hmac_key_id] stripped
  canonical   = json.dumps(content, sort_keys=True, default=str)
  message     = HMAC_KEY_ID + ":" + canonical + previous_hmac
  digest      = HMAC-SHA256(key=HMAC_KEY, msg=message).hexdigest()

  event.hmac          = digest        (64-char hex string)
  event.previous_hmac = prev_digest   (from preceding event, or GENESIS_HMAC)
  event.hmac_key_id   = HMAC_KEY_ID

Chain genesis

The first audit event in a chain uses the genesis sentinel as its previous_hmac:

GENESIS_HMAC = "0" * 64  # 64 zero characters

This sentinel is a fixed, well-known value (not secret). It establishes the root of the chain without requiring a special initialization event.

Key rotation

The hmac_key_id field (added in migration 049) enables HMAC key rotation without breaking chain verification:

1. Configure a new AUDIT_HMAC_KEY and AUDIT_HMAC_KEY_ID value (e.g., "v2").
2. New events are signed with the new key and tagged with the new hmac_key_id.
3. During verification, the verifier uses event.hmac_key_id (or "default" for pre-rotation events) to select the correct key for recomputation.

Key management

The HMAC key is stored in AUDIT_HMAC_KEY, listed in the platform’s SENSITIVE_SETTINGS (masked in logs and diagnostic endpoints). When AUDIT_HMAC_KEY is empty, HMAC chaining is silently disabled on the platform — events are still recorded but without hmac or previous_hmac values.

Sink configuration

The AUDIT_SINKS env var accepts a comma-separated list:

Sink	Description	Configuration
db	PostgreSQL audit_logs table	Default, always available
jsonl	Append-only JSONL file	AUDIT_JSONL_PATH (default: /var/log/arbitex/audit.jsonl)
webhook	HTTP POST to external URL	AUDIT_WEBHOOK_URL, AUDIT_WEBHOOK_TOKEN
splunk_hec	Splunk HTTP Event Collector	Splunk HEC configuration

Failed sink writes are captured in a dead-letter queue (DEAD_LETTER_MAX_SIZE = 1000, FIFO eviction on overflow) to prevent silent audit event loss.

---

Outpost audit events

AuditLogger

The outpost maintains its own independent HMAC chain stored in a local JSONL file (audit.jsonl). Unlike the platform, the outpost requires a valid AUDIT_HMAC_KEY — the process will refuse to start without one.

Configuration:

Env var	Default	Description
AUDIT_HMAC_KEY	(required)	HMAC signing key. Must be at least 32 characters. Startup fails if empty or too short.
AUDIT_BUFFER_PATH	audit_buffer/	Directory for audit.jsonl file.
MAX_AUDIT_BUFFER_ENTRIES	100000	Maximum entries in the audit buffer before rotation.
AUDIT_SYNC_INTERVAL_SECONDS	30	Interval for syncing events to the platform.

Outpost HMAC computation

The outpost uses a different HMAC formula than the platform:

GENESIS_HASH   = "0" × 64  (64 zero characters — same sentinel)

For each event:
  entry_data  = event dict (chain fields excluded)
  canonical   = json.dumps(entry_data, sort_keys=True, default=str,
                           separators=(",", ":"))
  message     = prev_hash + canonical
  digest      = HMAC-SHA256(key=AUDIT_HMAC_KEY, msg=message).hexdigest()

  event.hmac      = digest
  event.prev_hash = prev_digest

Cross-system formula difference

The platform formula is key_id + ":" + canonical + previous_hmac. The outpost formula is prev_hash + canonical. These are intentionally different implementations. Cross-system chain verification requires using the correct formula for each tier. The outpost also uses compact JSON separators ((",", ":")) while the platform uses default separators.

OCSF-compatible event structure

Each outpost audit entry follows the Open Cybersecurity Schema Framework (OCSF v1.1):

{
  "class_uid": 6003,
  "activity_id": 1,
  "severity_id": 1,
  "status_id": 1,
  "time": "2026-03-15T12:00:00.000Z",
  "time_epoch": 1773756000000,
  "metadata": {
    "version": "1.1.0",
    "product": { "name": "Arbitex Outpost", "vendor_name": "Arbitex" },
    "log_name": "outpost_audit"
  },
  "observer": {
    "type_id": 1,
    "name": "outpost-prod-01"
  },
  "event_id": "a1b2c3d4-...",
  "geoip": { "country": "US", "city": "..." },
  "trace_id": "abc123...",
  "span_id": "def456...",
  "synced": false,
  "hmac": "e5f6a7b8...",
  "prev_hash": "0000000000000000000000000000000000000000000000000000000000000000"
}

Field	Description
class_uid	OCSF class: 6003 (API Activity)
activity_id	1 (Create)
severity_id	1 (Informational)
status_id	1 (Success, HTTP < 400) or 2 (Failure, HTTP >= 400)
time	ISO 8601 UTC timestamp
time_epoch	Unix epoch in milliseconds
observer.name	Outpost ID from OUTPOST_ID env var
trace_id / span_id	OpenTelemetry correlation IDs (when available)
synced	Whether this event has been synced to the platform
hmac	HMAC-SHA256 hex digest
prev_hash	Previous event’s HMAC digest

Audit buffer rotation

When the JSONL file reaches MAX_AUDIT_BUFFER_ENTRIES (default 100,000):

1. The current audit.jsonl is moved to audit.jsonl.1 (overwriting any existing .1 file).
2. A new audit.jsonl is created.
3. The HMAC chain resets to GENESIS_HASH for the new file.

On startup, the AuditLogger reads the last line of the existing audit.jsonl and extracts its hmac field to resume the chain — providing continuity across process restarts without full chain replay.

DLP scan ring buffer

The outpost maintains a separate in-memory ring buffer (50 entries) specifically for DLP scan events, accessible via the admin API for quick operational visibility:

_SCAN_RING_BUFFER_SIZE = 50
_scan_ring_buffer: deque[dict] = deque(maxlen=50)

This ring buffer is separate from the JSONL audit buffer. It holds only DLP scan events and resets on process restart. It is not HMAC-chained.

Ring buffer stats tracking:

Stat	Type	Description
total_scans	int	Cumulative DLP scans since startup
total_blocked	int	Requests blocked by DLP rules
total_redacted	int	Requests with content redacted
total_scan_duration_ms	float	Cumulative scan time
entity_type_counts	Counter	Detection counts by entity type
action_distribution	Counter	Counts by action taken (ALLOW, BLOCK, etc.)
recent_scan_timestamps	deque(600)	5-minute rolling window for rate calculation

Admin API endpoints for the ring buffer:

Endpoint	Description
GET /admin/audit/recent	Last 50 DLP scan events from ring buffer
GET /admin/audit/stats	Aggregate DLP scan statistics since startup

---

Outpost-to-platform sync

Outpost audit events are periodically synced to the platform for durable storage and centralized visibility.

Sync mechanism

The outpost’s AuditSyncWorker reads unsent events from audit.jsonl and sends them in batches to the platform:

• Endpoint: POST /v1/internal/outpost-audit-sync
• Transport: mTLS (outpost client certificate → platform CA)
• Batch size: 50 events per request
• Interval: AUDIT_SYNC_INTERVAL_SECONDS (default 30s)
• Payload: { outpost_id, batch_size, events }
• Headers: X-Outpost-ID for routing

The platform receives synced events and stores them in the audit_logs table with source="outpost" and the originating outpost_id. After storage, accepted events are forwarded to SIEM sinks via the connector registry.

Graceful degradation

When the platform is unreachable, the outpost’s circuit breaker opens and events are written to a local SQLite queue instead of being dropped:

CREATE TABLE local_audit_queue (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    event_json  TEXT    NOT NULL,
    queued_at   REAL    NOT NULL,
    retry_count INTEGER NOT NULL DEFAULT 0
);

Admin API for queue management:

Endpoint	Method	Description
GET /admin/audit-queue/status	GET	Queue stats: count, oldest_queued_at, disk_usage_bytes, degradation_mode
POST /admin/audit-queue/flush	POST	Trigger immediate queue flush to platform
DELETE /admin/audit-queue/purge	DELETE	Purge all queued events (requires X-Purge-Confirm: yes header)

When the circuit breaker closes (platform reachable again), queued events are drained in batches and synced to the platform. Events are retried with an incrementing retry_count.

---

Chain verification

Platform verification endpoint

POST /api/v1/admin/audit/verify

Authentication: Admin JWT required.

Behavior:

1. Fetches all AuditLog rows ordered by created_at ASC.
2. Reconstructs event dicts from the stored model fields.
3. Calls verify_chain(events, key) which checks:
   • The first event’s previous_hmac equals GENESIS_HMAC.
   • Each event’s stored hmac matches the recomputed HMAC-SHA256 digest.
   • Each event’s previous_hmac matches the preceding event’s hmac.
4. Uses event.hmac_key_id (falling back to "default" for pre-rotation entries) to select the correct key.

Response:

{
  "valid": true,
  "total_entries": 15847,
  "errors": []
}

When a break is detected:

{
  "valid": false,
  "total_entries": 15847,
  "errors": [
    "Event 2834: HMAC mismatch — stored=a1b2... computed=c3d4...",
    "Event 2835: previous_hmac does not match prior event hmac"
  ]
}

Error cases:

• HTTP 400 if AUDIT_HMAC_KEY is not configured.

What “valid chain” means

A valid chain guarantees:

1. Completeness — no events have been deleted from the sequence. Each event’s previous_hmac creates an unbroken link to its predecessor.
2. Integrity — no event has been modified after creation. Any change to a signed field would produce a different HMAC digest.
3. Ordering — events are in the correct chronological sequence. Reordering events would break the previous_hmac linkage.
4. Non-repudiation — the HMAC key holder (the platform) can prove the chain was produced by an entity with knowledge of the key.

A broken chain does not necessarily indicate malicious tampering — it can also result from:

• Database restoration from a backup that excluded recent entries
• HMAC key rotation without proper migration
• A bug in event serialization (field ordering, type coercion)

Outpost verification

Tip

The outpost provides two chain verification endpoints: GET /admin/audit/verify (compact response with valid, events_checked, first_broken_at) and GET /admin/api/audit/verify (extended response with verified, total_events, first_event_id, last_event_id, chain_valid, broken_at_event). Both verify the HMAC chain integrity of the in-memory DLP scan ring buffer (50-event capacity). When AUDIT_CHAIN_ENABLED=false, both return valid with a note. See Outpost Admin API — Audit & SIEM for full schemas. For offline verification, operators can also read audit.jsonl and recompute the HMAC chain programmatically using the documented formula.

The outpost’s JSONL-based chain can be verified offline by reading the file line by line and recomputing:

# Each line is a JSON object with hmac and prev_hash fields
# Verify: for each pair of consecutive entries,
#   entry[n].prev_hash == entry[n-1].hmac
# And recompute each HMAC using the outpost formula

---

Anomaly detection

The platform includes three automated anomaly detection checks that run against the audit log:

Check	Trigger	Parameters
Volume spike	Current hourly request rate exceeds 3× the 7-day rolling average	VOLUME_SPIKE_MULTIPLIER = 3.0, ROLLING_WINDOW_DAYS = 7
Off-hours access	Admin-role users performing actions during quiet hours	Default window: 22:00–06:00 UTC
Unusual model usage	First-time (user_id, model_id) pair observed in the evaluation window	Flags novel model access patterns

These checks produce anomaly records that can be surfaced in dashboards and SIEM integrations. They support SOC 2 CC7.2 (anomaly detection) compliance requirements.

---

Compliance mapping

HIPAA — audit requirement

HIPAA requires that covered entities maintain audit controls that record and examine access to electronic protected health information (ePHI). The Arbitex audit trail satisfies this by:

• Recording every AI gateway interaction with user identity, timestamp, and action
• Maintaining HMAC chain integrity to prove logs have not been tampered with
• Supporting configurable retention (platform audit events are durable in PostgreSQL)
• Providing the /api/v1/admin/audit/verify endpoint for on-demand integrity verification

Note

HIPAA audit log retention requirements vary. The HIPAA Security Rule requires audit controls but does not specify a retention period. The HHS enforcement practice and most compliance frameworks recommend 6 years. Configure your PostgreSQL retention policy accordingly.

SOC 2 — CC7.2 anomaly detection

SOC 2 Trust Services Criteria CC7.2 requires organizations to monitor system components for anomalous activity. Arbitex supports this through:

• Volume spike detection — identifies unusual request patterns
• Off-hours access monitoring — flags admin actions outside business hours
• Unusual model usage — detects first-time model access patterns
• HMAC chain verification — proves log integrity for auditor review

PCI-DSS — Requirement 10.2

PCI-DSS Requirement 10.2 mandates audit trails for cardholder data access. Arbitex’s DLP pipeline detects credit card numbers (PAN) and the audit trail records:

• Every scan event with entity types detected (including CREDIT_CARD)
• The action taken (BLOCK, REDACT, LOG_ONLY)
• User identity and timestamp
• HMAC-signed chain proving completeness

Using HMAC to prove log integrity to auditors

For compliance audits, demonstrate log integrity by:

1. Run chain verification:

   curl -X POST https://api.arbitex.ai/api/v1/admin/audit/verify \
     -H "Authorization: Bearer <admin-jwt>"

2. Present the response — a valid: true result with total_entries count proves the complete chain is intact.

3. Explain the mechanism — each event is cryptographically linked to its predecessor using HMAC-SHA256. Tampering with any event breaks the chain from that point forward.

4. Key custody — document who has access to the AUDIT_HMAC_KEY. The key should be stored in a secrets manager (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) with access logging.

---

Limitations

In-memory ring buffer

The DLP scan ring buffer holds only the last 50 events in memory. It is designed for quick operational checks, not for long-term analysis:

• Volatile — resets on process restart
• Not HMAC-chained — separate from the JSONL audit chain
• Not synced — exists only in the outpost process memory

For durable DLP scan history, use the JSONL audit buffer or the platform’s synced audit logs.

Outpost restart

When the outpost process restarts:

• The DLP scan ring buffer is cleared (50 events lost)
• The DLP scan stats counters reset to zero
• The JSONL audit chain resumes from the last entry in audit.jsonl (chain continuity preserved)
• The SQLite local queue is durable and survives restarts

Buffer rotation

When the JSONL audit buffer rotates (audit.jsonl → audit.jsonl.1):

• Only one rotated file is kept (.1 overwrites any prior .1)
• The HMAC chain resets to GENESIS_HASH in the new file
• Historical events in .1 are still valid but form a separate chain

Air-gap mode

In air-gap deployments (OUTPOST_AIRGAP=true):

• No sync to platform — events stay in local JSONL only
• The SQLite local queue is not used (no sync target)
• Operators must export and archive audit.jsonl manually for retention compliance
• Policy bundles are loaded from disk at startup (AIRGAP_POLICY_PATH/policy_bundle.json) and cannot be hot-reloaded via admin API — process restart required

---

Integration with OCSF

The Open Cybersecurity Schema Framework (OCSF) provides a standardized event format for security telemetry. Arbitex audit events map to OCSF as follows:

Class mapping

OCSF class	Class UID	Arbitex usage
API Activity	6003	All outpost audit events (DLP scans, inference requests)

Field mapping

OCSF field	Arbitex field	Notes
class_uid	6003	Fixed: API Activity
activity_id	1	Fixed: Create
severity_id	1	Informational (default)
status_id	1 or 2	1 = Success (HTTP < 400), 2 = Failure (HTTP >= 400)
time	time	ISO 8601 UTC
time_epoch	time_epoch	Unix epoch milliseconds
metadata.version	—	OCSF schema version: "1.1.0"
metadata.product.name	—	"Arbitex Outpost"
metadata.log_name	—	"outpost_audit"
observer.type_id	—	1 (Endpoint)
observer.name	OUTPOST_ID	Outpost instance identifier

Platform events and OCSF

Platform audit events stored in PostgreSQL do not natively use OCSF field names. When synced from the outpost, the platform preserves the original OCSF structure in the event metadata. For direct platform events, the mapping is:

Platform field	OCSF equivalent
id (UUID)	event_id
action	activity_name
created_at	time
user_id	actor.user.uid
tenant_id	cloud.org.uid
src_ip	src_endpoint.ip
dst_ip	dst_endpoint.ip
model_id	api.request.resource_uid
provider	api.service.name
metadata	unmapped (OCSF extension)

SIEM forwarding

Outpost events synced to the platform are forwarded to configured SIEM sinks (Splunk HEC, webhook) with the OCSF structure preserved. This enables correlation with other OCSF-compatible security tools without field translation.

---

Configuration reference

Platform environment variables

Variable	Default	Description
AUDIT_HMAC_KEY	"" (disabled)	HMAC-SHA256 signing key. When empty, chain integrity is disabled.
AUDIT_HMAC_KEY_ID	"default"	Key version identifier for rotation support.
AUDIT_SINKS	"db"	Comma-separated sink list: db, jsonl, webhook, splunk_hec.
AUDIT_JSONL_PATH	/var/log/arbitex/audit.jsonl	Path for JSONL sink output.
AUDIT_WEBHOOK_URL	""	Webhook sink destination URL.
AUDIT_WEBHOOK_TOKEN	""	Bearer token for webhook authentication.

Outpost environment variables

Variable	Default	Description
AUDIT_HMAC_KEY	(required)	HMAC-SHA256 signing key. Must be ≥ 32 characters. Startup fails if empty.
AUDIT_BUFFER_PATH	audit_buffer/	Directory for audit.jsonl.
MAX_AUDIT_BUFFER_ENTRIES	100000	Maximum entries before buffer rotation.
AUDIT_SYNC_INTERVAL_SECONDS	30	Sync interval to platform.
AUDIT_QUEUE_DB_PATH	audit_queue/audit_queue.db	SQLite local queue path (degradation mode).

---

Related documentation

• Audit Data Model Reference — complete field-level schema
• Audit log export — admin API for searching and exporting audit entries
• Security architecture — overall security design
• Outpost architecture — outpost component overview
• Troubleshooting — audit chain debugging