IR-7: HMAC Chain Break Detected

An HMAC chain break means the audit trail’s cryptographic integrity has failed for one or more entries. The platform links every audit event to the previous one via HMAC-SHA256, anchored to a known genesis value. A break indicates either unauthorized database modification, data corruption, or deliberate evidence destruction.

For background on the chain construction and invariants, see Audit Chain Integrity. For the full audit log management guide, see Audit Log Management.

---

1. Detection

An HMAC chain break surfaces through one of the following signals:

Signal	Source	Notes
Scheduled chain verification job reports errors	Cron job calling POST /api/v1/admin/audit/verify	Runs hourly by default
Manual verify returns valid: false	Direct API call	Includes per-event error list
SOC 2 compliance check fails audit integrity control	Compliance dashboard	Quarterly at minimum
Monitoring alert from audit.chain_break event	Platform alerting	Emitted by verification job
Unexpected database modification detected	Postgres slow query log or WAL	Direct SQL access bypasses HMAC write path

Confirm the break and capture the error payload:

# Run chain verification for the calling admin's org
curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, total_entries, errors}'

A broken chain returns:

{
  "valid": false,
  "total_entries": 48312,
  "errors": [
    "Event 1234: HMAC mismatch — stored abc123, computed def456",
    "Event 1235: previous_hmac linkage broken — expected abc123, found 000000"
  ]
}

The error messages identify the exact break point by event sequence number. Record these values immediately — they are the primary forensic lead.

---

2. Severity classification

Condition	Severity
Break detected, no correlation with other incidents	HIGH
Break window overlaps with another active incident	CRITICAL
Break appears in multiple org windows	CRITICAL
Break confirmed as deliberate tampering (DB log evidence)	CRITICAL
Single-event break consistent with key rotation mishandling	MEDIUM

Escalate to CRITICAL immediately if:

• The break window corresponds to an active DLP bypass, exfiltration, or compromise incident — the audit evidence for that incident may be corrupted
• Database logs show a direct UPDATE or DELETE on the audit events table from outside the application
• More than one org’s chain is broken within the same time window

---

3. Immediate actions (first 15 minutes)

Complete in order. Do not compact or delete any audit entries before investigation is complete.

Step	Action	Command
1	Declare incident	POST /api/staff/incident/declare
2	Freeze audit compaction	POST /api/staff/emergency/audit/freeze
3	Snapshot the database	Out-of-band — Azure PG snapshot or pg_dump
4	Export audit trail around break	POST /api/v1/admin/audit/export
5	Begin investigation	Section 4

Step 1 — Declare incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/declare" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "HMAC audit chain break — events ~<break_event_id>",
    "severity": "HIGH",
    "playbook": "IR-7",
    "notes": "Break detected at event <break_event_id>. Freezing audit compaction."
  }' | jq '{incident_id: .id, ic_expires: .ic_window_expires}'

Save the returned incident_id.

Step 2 — Freeze audit log compaction:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/audit/freeze" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-7: HMAC chain break — preserving evidence, halting compaction"
  }' | jq '{frozen: .frozen, freeze_expires: .expires_at}'

The freeze suspends audit log compaction and retention enforcement. Evidence cannot be deleted while the freeze is active. The freeze has a maximum duration (check response expires_at) — extend via POST /api/staff/incident/{id}/extend if the investigation runs long.

Step 3 — Database snapshot:

Take an out-of-band snapshot immediately. The application snapshot preserves the exact state of the audit table at the time of discovery. Use Azure Portal > Flexible Server > Restore Point, or run:

# On a host with pg_dump access
pg_dump -h <db_host> -U <db_user> -t audit_events -Fc \
  -f "ir7-audit-snapshot-$(date +%Y%m%dT%H%M%S).dump" arbitex

Step 4 — Export audit trail around the break:

Export a window large enough to include at least 100 events before and after the reported break point:

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/export" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "start_date": "<1h_before_break_date>",
    "end_date": "<1h_after_break_date>",
    "format": "json",
    "include_metadata": true
  }' -o "ir7-audit-export-<incident_id>.json"

Store this export to secure evidence storage before doing anything else.

---

4. Investigation

The goal is to determine whether the break is due to corruption (accidental) or tampering (deliberate), and to identify the exact boundary of compromised entries.

Identifying the break point

Three HMAC invariants must hold for the chain to be valid:

Invariant	Check
Genesis	First event must have previous_hmac = 64 zero hex chars
HMAC correctness	Recomputed HMAC must match stored hmac field
Chain linkage	Each event’s previous_hmac must match the preceding event’s hmac

Narrow the break window using bisecting verify calls:

# Verify the full chain for the calling admin's org
curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, errors}'

Checking for direct database modification

Look for SQL statements modifying the audit table outside the application write path:

-- In Postgres: check WAL or audit log (pg_audit extension)
SELECT statement_timestamp, usename, application_name, query
FROM pg_stat_activity
WHERE query ILIKE '%audit_events%'
  AND query NOT ILIKE '%SELECT%';

If pg_audit is enabled, check the audit log for non-application UPDATE or DELETE on audit_events:

grep -E "(UPDATE|DELETE).*audit_events" /var/log/postgresql/pgaudit.log | tail -100

Checking for AUDIT_HMAC_KEY changes

An AUDIT_HMAC_KEY change without re-signing breaks the chain from the point of the change forward. Check for recent key rotation:

# Check Azure Key Vault audit log for key operations
az keyvault key list-versions --vault-name <vault_name> --name audit-hmac-key \
  --query "[*].{version:name, created:attributes.created, updated:attributes.updated}"

If the key was rotated recently, this may explain an accidental break rather than deliberate tampering. Cross-reference the key rotation timestamp with the break event timestamp.

Verifying chain integrity across orgs

Determine if the break is isolated to one org or platform-wide:

# Verify chain integrity — the endpoint verifies the calling admin's org.
# For multi-org checks, use a staff token scoped to each org and run sequentially:
for ORG_ID in $(curl -s "https://api.arbitex.ai/api/v1/admin/orgs" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq -r '.orgs[].id'); do
  RESULT=$(curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
    -H "Authorization: Bearer $STAFF_TOKEN" \
    -H "X-Org-Id: ${ORG_ID}" \
    | jq -r '.valid')
  echo "${ORG_ID}: ${RESULT}"
done

A platform-wide break (multiple orgs) strongly suggests a database-level modification or key rotation event, not an isolated tampering of a single org’s events.

---

5. Containment

Action	When required	Command
Keep audit freeze active	Always, until investigation complete	Monitor freeze_expires, extend if needed
Preserve DB snapshot to cold storage	Always	Move dump to isolated secure storage
Extend incident window	Investigation running past 1h IC window	POST /api/staff/incident/{id}/extend
Escalate to CRITICAL	Tampering confirmed or overlaps other incident	Update incident severity, notify security lead

Extend the IC window if needed:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/extend" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"extend_hours": 1, "reason": "Investigation ongoing — pending DB log analysis"}'

Do not:

• Compact or truncate any audit events in the break window
• Rotate the AUDIT_HMAC_KEY until you understand whether this was the cause
• Delete or overwrite the database snapshot

---

6. Recovery

The correct recovery path depends on the root cause.

Path A: Accidental break due to key rotation

If the break was caused by an AUDIT_HMAC_KEY rotation (key changed without re-signing existing entries):

# Re-sign the affected segment using the new key
# This is a platform maintenance operation — run via the platform admin tooling
# DO NOT do this via ad-hoc SQL; use the provided re-sign CLI command:
python -m arbitex.tools.audit_resign \
  --from-event <first_broken_event_id> \
  --to-event <last_broken_event_id> \
  --confirm

After re-signing, verify the chain is intact:

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, total_entries, errors}'

Path B: Tampering confirmed

If the break is confirmed as deliberate (direct DB modification, entries deleted or overwritten):

1. Preserve the broken chain as forensic evidence — do not modify it.
2. Document the gap: record which event IDs are affected, what was changed, and the approximate timestamp.
3. The audit evidence for the affected window is compromised and cannot be relied upon in legal or compliance proceedings.
4. Begin a new chain segment from the last verified-good event using the platform’s chain-resume tooling. Mark the gap in the compliance record.

# Verify the current chain state for the calling admin's org
curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, total_entries}'

Verify platform health after recovery

curl -s "https://api.arbitex.ai/readyz" | jq '.'
curl -s "https://api.arbitex.ai/health/deep" | jq '{status, db, redis}'

Unfreeze the audit log only after chain integrity is verified:

# Audit freeze auto-expires; to release early contact the platform team
# Verify: confirm no compaction has been re-triggered on the affected window

---

7. Communication

Internal notification (immediate — within 15 minutes of detection)

Notify the security lead and compliance officer:

Subject: [IR-7] HMAC audit chain break detected — investigation in progress

Incident ID: [INCIDENT_ID]
Detected: [ISO_TIMESTAMP]
Break point: Event [BREAK_EVENT_ID] (approximately [TIMESTAMP])
Scope: [Isolated to org X / Platform-wide across Y orgs]
Status: Audit compaction frozen, evidence snapshot taken, investigation in progress.

Root cause determination expected: [TIME_ESTIMATE]

Contact: [IC_NAME] — [CONTACT]

Escalation to legal (tampering confirmed)

If deliberate tampering is confirmed, notify Arbitex legal within 1 hour:

Subject: [URGENT] Audit trail tampering confirmed — legal review required

Incident ID: [INCIDENT_ID]
Finding: Audit log entries were modified or deleted outside the application write path between [START] and [END].
Scope: [N] events affected across [orgs / platform].
Impact: Audit evidence for this window is unreliable and cannot be used in legal proceedings without disclosure of the gap.
Evidence preserved: [YES/NO — describe snapshot location]

Immediate action requested: legal hold on all related evidence.

Customer notification (tampering confirmed, customer org affected)

If a specific org’s audit trail was tampered with, notify that customer:

Subject: [Security Notice] Audit trail integrity event affecting your Arbitex account

We are writing to inform you of an integrity event affecting the audit log for your Arbitex organization.

Between [START_TIMESTAMP] and [END_TIMESTAMP], audit log entries for your organization were affected by a chain integrity failure. We have preserved all available evidence and are conducting a full investigation.

What this means for you:
- Audit exports for this period may reflect incomplete data.
- We will provide a full audit export and our investigation findings within 72 hours.

Your compliance team should note this gap when conducting any SOC 2 or regulatory audit covering this period.

Incident reference: [INCIDENT_ID]
Contact: [email protected]

---

8. Post-incident

Evidence preservation checklist:

Item	Status
Database snapshot saved to secure cold storage	[ ]
Audit export (break window ±1 hour) archived	[ ]
Verify API response (with error list) recorded	[ ]
Database modification logs captured (WAL / pg_audit)	[ ]
Key Vault audit log for HMAC key operations captured	[ ]
Root cause determination documented	[ ]
Chain re-signed or gap formally documented	[ ]
Compliance record updated with gap disclosure	[ ]

Close the incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/close" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Chain break identified and resolved — <accidental/tampering>. Evidence preserved. Chain re-signed / gap documented.",
    "root_cause": "<key_rotation_error / db_modification / tampering / corruption>",
    "data_exfiltrated": false
  }'

Lessons learned template:

• Was the scheduled verification job running at the expected frequency?
• How long between the break occurring and detection?
• Was there a database access audit trail in place (pg_audit / Key Vault logs)?
• Was AUDIT_HMAC_KEY rotation coordinated with the re-sign procedure?
• Should verification frequency be increased?

Related playbooks:

• If the break correlates with unauthorized database access: follow IR-5: Database Breach Suspected
• If the break overlaps with a compromised tenant or insider event: escalate per IR-12: Insider Threat