IR-10: Mass Credential Breach

Severity: CRITICAL

Scope: CredInt (Credential Intelligence) detects a high-volume spike of known-compromised credentials appearing in AI gateway requests across multiple organizations. This pattern indicates a major third-party breach where attackers are testing stolen credentials against the platform at scale.

IC role required for: tenant isolation, bulk credential revocation, audit freeze.

Related documentation: Credential Intelligence — Credential Management — Security Operations — Audit Log Verification

---

1. Detection

A mass credential breach is characterized by a sudden, correlated spike in CredInt hits across multiple organizations rather than isolated incidents.

Signal	Source	Notes
High-volume credint_hit audit events across multiple org IDs	CredInt audit events	Distinguish from routine single-org hits — look for breadth
frequency_bucket: "critical" hits spiking within a short window	CredInt event metadata	Critical bucket indicates credential appears frequently in breach databases
CredInt circuit breaker state: open	/health/deep CredInt status	Open state means checks fail open — compromised credentials pass through undetected
SIEM correlation: many distinct credentials from few source IPs	SIEM / audit log aggregation	Credential stuffing pattern — attacker testing a breach dump
Abnormal auth success rates across orgs	Platform metrics / Grafana dashboard	Success after credint_hit means the credential is valid on Arbitex

Query for CredInt hit volume across all orgs:

# Count credint_hit events in the last 60 minutes, grouped by org
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=credint_hit&created_after=<60_min_ago_iso>&limit=1000" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq 'group_by(.org_id) | map({org_id: .[0].org_id, hit_count: length}) | sort_by(-.hit_count)'

Check CredInt circuit breaker state:

curl -s "https://api.arbitex.ai/health/deep" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '.credint | {state, failure_count, last_failure, reset_at}'

If state is "open", the circuit breaker has tripped (3 consecutive failures). In this state, CredInt checks are bypassed and all credentials pass through unchecked. This is fail-open by design for availability, but it means the true scope of compromised credentials may be larger than the hit count suggests. Note this in your investigation scope.

---

2. Severity classification

Condition	Severity
Single credint_hit on one org, isolated credential	LOW — handled per IR-2
Multiple hits across 2-5 orgs within 30 minutes	HIGH
High-volume hits across 6+ orgs — possible mass breach event	CRITICAL
CredInt circuit breaker open during the event	CRITICAL — unknown additional exposure
Confirmed successful authentication with breached credential	CRITICAL

Escalate to CRITICAL and activate IC if:

• CredInt hits span 6 or more organizations
• Any breached credential resulted in a successful API call (check audit events post-credint_hit)
• CredInt circuit breaker is open and hit volume is high

---

3. Immediate actions (first 15 minutes)

Do not skip steps. The ordering matters — declare first so all subsequent actions are correlated to the incident ID.

Step	Action	Command / UI
1	Declare incident	POST /api/staff/incident/declare
2	Verify CredInt service health and circuit breaker state	GET /health/deep
3	Freeze audit log to preserve evidence	POST /api/staff/emergency/audit/freeze
4	Identify all affected org IDs from CredInt events	Query audit trail
5	For each affected org: revoke sessions for affected users	DELETE /api/v1/admin/users/{user_id}/sessions
6	Begin forced credential rotation for confirmed-compromised credentials	DELETE /api/v1/admin/credentials/{id}

Step 1 — Declare incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/declare" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Mass credential breach — CredInt mass hit event",
    "severity": "CRITICAL",
    "playbook": "IR-10"
  }' | jq '{incident_id: .id, ic_expires: .ic_window_expires}'

Save the incident_id — all subsequent calls must reference it.

Step 2 — Verify CredInt service health:

curl -s "https://api.arbitex.ai/health/deep" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{overall: .status, credint: .credint}'

If credint.state is "open", note this in the incident record. The circuit breaker opens after 3 consecutive CredInt backend failures within a 60-second window and resets after 60 seconds of healthy responses. A mass breach event generating high request volume may itself cause CredInt backend overload and trigger the circuit breaker — treat any open state during a mass breach as a scope-widening factor.

Step 3 — Freeze audit log:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/audit/freeze" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-10: Mass credential breach — preserving audit evidence"
  }'

Step 4 — Identify affected organizations:

# Get the full list of org IDs with credint_hit events in the event window
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=credint_hit&created_after=<event_start_iso>&limit=2000" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '[.events[] | .org_id] | unique | sort'

Record the list of affected org IDs. This is your working scope for the rest of the incident.

---

4. Investigation

For each affected org, determine: (a) which users had compromised credentials detected, (b) whether any of those credentials were used successfully for unauthorized access, and (c) the likely breach source.

Aggregate CredInt hits per org to identify most-affected organizations:

curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=credint_hit&created_after=<event_start_iso>&limit=2000" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq 'group_by(.org_id) | map({
      org_id: .[0].org_id,
      hit_count: length,
      frequency_buckets: (group_by(.metadata.frequency_bucket) | map({bucket: .[0].metadata.frequency_bucket, count: length})),
      unique_sha1_prefixes: ([.[].metadata.sha1_prefix] | unique | length)
    })'

The sha1_prefix field contains the first 8 hex characters of the SHA-1 hash of the credential. Multiple hits sharing the same sha1_prefix represent the same credential being used repeatedly. The cleartext credential is never logged.

Identify whether breached credentials were used successfully after detection:

# For a specific org, find auth.login_success or gateway.request events
# that occurred AFTER a credint_hit for the same user_id
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?org_id=<org_uuid>&action=auth.login_success&created_after=<event_start_iso>" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '.events[] | {timestamp, user_id, source_ip, metadata}'

Cross-reference the user_id from successful auth events against the user_id from credint_hit events in the same window. Any overlap indicates the breached credential was also valid on the platform and may have been used by an attacker.

Correlate source IPs across CredInt events to identify attacker infrastructure:

# Find IPs sending the highest volume of requests with credint hits
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=credint_hit&created_after=<event_start_iso>&limit=2000" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq 'group_by(.source_ip) | map({ip: .[0].source_ip, count: length}) | sort_by(-.count) | .[0:20]'

A small number of IPs sending a high volume of credint_hit events across many orgs is characteristic of a credential stuffing attack. These IPs are candidates for WAF block rules (see containment).

Verify audit chain integrity for the event window:

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, total_entries, errors}'

---

5. Containment

Scale containment to confirmed scope. Work through each affected org.

Per-org containment steps:

Action	When	Command
Revoke sessions for affected users	All affected orgs	DELETE /api/v1/admin/users/{user_id}/sessions
Revoke the breached credential	Credential identified	DELETE /api/v1/admin/credentials/{id}
Force credential rotation (user-initiated)	Credential rotation preferred over full revoke	POST /api/v1/admin/credentials/{id}/rotate
Bulk credential revocation for org	Scope unclear or multiple credentials hit	POST /api/staff/emergency/credentials/revoke-all
Tenant isolation	Confirmed unauthorized access occurred	POST /api/staff/emergency/tenant/isolate

Revoke sessions for a specific user:

curl -s -X DELETE "https://api.arbitex.ai/api/v1/admin/users/<user_id>/sessions" \
  -H "Authorization: Bearer $STAFF_TOKEN"

Revoke a specific compromised credential:

curl -s -X DELETE "https://api.arbitex.ai/api/v1/admin/credentials/<credential_id>" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{status, revoked_at}'

Bulk revoke all credentials for an org where unauthorized access is confirmed:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/credentials/revoke-all" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "org_id": "<org_uuid>",
    "incident_id": "<incident_id>",
    "reason": "IR-10: Mass credential breach — bulk revocation, unauthorized access confirmed"
  }' | jq '{revoked_count: .revoked, credential_ids: .ids}'

Block known attacker IPs at the edge (Cloudflare WAF):

Provide the aggregated attacker IP list to the network/security team for WAF block rule creation. Do not attempt to block IPs via platform controls — IP blocking for credential stuffing attacks must be applied at the Cloudflare edge layer to be effective.

---

6. Recovery

Begin recovery only after containment is complete for each affected org and the attack traffic has subsided.

Step	Action	Verification
1	Verify CredInt circuit breaker has closed	GET /health/deep — credint.state == "closed"
2	Issue replacement credentials to affected users	POST /api/v1/admin/credentials/{id}/rotate
3	Notify each affected org to rotate credentials in their source systems	Per-org communication (section 7)
4	Restore tenant access for any isolated orgs	POST /api/staff/emergency/tenant/restore
5	Verify platform health	GET /readyz, GET /health/deep
6	Confirm CredInt is processing normally post-incident	Monitor credint_hit event rate — should return to baseline

Verify CredInt circuit breaker state before restoring normal operations:

curl -s "https://api.arbitex.ai/health/deep" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '.credint | {state, consecutive_failures: .failure_count}'

state: "closed" with failure_count: 0 confirms CredInt is operating normally.

Rotate a credential to issue a fresh secret while preserving the credential record:

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/credentials/<credential_id>/rotate" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"grace_period_seconds": 0}'

Setting grace_period_seconds: 0 ensures the old (breached) secret is immediately invalid. Provide the new secret to the key owner via a secure channel — not email.

Restore isolated tenant:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/tenant/restore" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "org_id": "<org_uuid>",
    "incident_id": "<incident_id>",
    "reason": "IR-10: Credentials rotated, breach contained, customer notified"
  }'

Platform health check:

curl -s "https://api.arbitex.ai/readyz" | jq '.'
curl -s "https://api.arbitex.ai/health/deep" | jq '{status, db, redis, credint}'

---

7. Communication

Notify each affected org individually. Do not share details of other organizations in any notification — each org receives only information about their own users and credentials.

Per-org initial notification (within 30 minutes of identification)

Subject: [SECURITY ALERT] Compromised credentials detected in your Arbitex account

We have detected that one or more credentials associated with your Arbitex
organization were found in an external breach database. This means the
credential values were exposed in a data breach of a third-party service.

Arbitex detected these credentials via our Credential Intelligence service,
which checks credentials against known breach corpora without retaining or
logging credential plaintext.

What we have done:
- Terminated active sessions for affected users in your organization
- Revoked the affected credential(s) to prevent unauthorized access
- Frozen your audit log to preserve evidence of any unauthorized activity

What you need to do:
1. Contact our security team to receive replacement credentials.
2. Audit the source system where these credentials were created — the
   exposure is in your environment, not in Arbitex systems.
3. Check for credential reuse across other services.

Affected credential prefix(es): [KEY_PREFIXES]
Affected user(s): [USER_EMAILS]
Detection time: [ISO_TIMESTAMP]
Incident reference: [INCIDENT_ID]

Contact: [email protected]

Internal escalation to legal (if scale triggers regulatory requirements)

If more than 10 organizations are affected, or if any affected org operates under GDPR, HIPAA, or other breach notification requirements, notify Arbitex legal within 1 hour of declaring CRITICAL. Provide:

• Incident ID
• Number of affected organizations
• Estimated number of affected user accounts
• Whether any breached credential resulted in confirmed unauthorized access
• Jurisdictions of affected orgs (for notification timing requirements)

Audit export for a specific org

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/export" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "org_id": "<org_uuid>",
    "start_date": "<event_start_date>",
    "end_date": "<recovery_complete_date>",
    "format": "json",
    "include_metadata": true
  }' -o "incident-<incident_id>-org-<org_uuid>-audit.json"

---

8. Post-incident

Evidence preservation checklist:

Item	Status
Audit export (full event window, all affected orgs) saved to secure storage	[ ]
HMAC chain verification result recorded	[ ]
CredInt circuit breaker state at time of detection documented	[ ]
Attacker IP list captured and forwarded to network team	[ ]
List of all affected orgs, affected users, and credential IDs recorded	[ ]
For each org: confirmed whether unauthorized access occurred	[ ]
Incident timeline document created	[ ]

Close the incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/close" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Breached credentials revoked, sessions terminated, affected orgs notified, replacement credentials issued",
    "root_cause": "Third-party breach of credential source (not Arbitex systems) — CredInt detected credentials in external corpus",
    "data_exfiltrated": false
  }'

Lessons learned template:

• What was the time between the breach event start and CredInt detection? Was the circuit breaker open at any point, and if so for how long?
• Were any breached credentials used successfully for API calls before revocation?
• Did per-org notifications go out within 30 minutes? If not, what caused the delay?
• Is the WAF IP block rule process documented and fast enough to respond during a credential stuffing campaign?
• Should rate limiting be tightened for requests that generate credint_hit events to slow stuffing attacks?

Related playbooks:

• If a specific org had confirmed unauthorized access: follow IR-1: Compromised Tenant Account
• If a specific API key was used for admin operations: follow IR-2: Compromised API Key