IR-11: Supply Chain Attack

Severity: CRITICAL

Scope: A dependency used by one or more Arbitex services has been found to contain malicious code — injected into an npm package, PyPI package, or Go module. Affected repos: platform (FastAPI + React + PG + Redis), cloud (FastAPI + React + PG), outpost (Go), staff (FastAPI + React + FIDO), core (Python library), www (Astro), docs (Markdown/Astro).

IC role required for: emergency kill switch, audit freeze, bulk credential revocation if runtime compromise is confirmed.

Related documentation: Security Operations — Audit Log Verification — Credential Management — Key Rotation

---

1. Detection

A supply chain compromise may be discovered through any of the following channels:

Signal	Source	Confidence
GitHub Dependabot / npm audit alert	Automated scanning in CI	HIGH — if CRITICAL severity
npm audit --omit=dev returns CRITICAL CVE	Manual sprint-close audit	HIGH
pip-audit or safety flags a PyPI package	Python dependency scan	HIGH
Published security advisory for a package Arbitex uses	NVD, package registry, security mailing list	HIGH
Behavioral anomaly: unexpected outbound network connections	Runtime monitoring	HIGH — possible active exfiltration
Community report of package compromise	Security researcher, upstream maintainer	HIGH
File system changes in unexpected locations during build	Build log analysis	MEDIUM — investigate further
Container image behavioral anomaly post-deploy	Runtime monitoring	MEDIUM

Run dependency audit across affected repos immediately upon detection:

# Node/npm repos (platform frontend, cloud frontend, staff frontend, www, docs)
cd <repo_path> && npm audit --omit=dev 2>/dev/null; echo "Exit: $?"

# Python repos (platform backend, cloud backend, staff backend, core, redteam)
cd <repo_path> && pip-audit --require-hashes -r requirements.txt 2>/dev/null

# Go repo (outpost)
cd <repo_path> && go mod verify 2>/dev/null; echo "Exit: $?"

Note the package name, affected version range, and CVE ID.

---

2. Severity classification

Condition	Severity
Vulnerability in a dev-only dependency, no runtime impact	LOW — patch next sprint
MODERATE/HIGH CVE in a runtime dependency	HIGH — patch within 24 hours
CRITICAL CVE in a runtime dependency	CRITICAL
Confirmed malicious code injection into a package (not just vulnerability)	CRITICAL
Malicious payload confirmed to have executed in build or runtime environment	CRITICAL
Evidence of data exfiltration from build or runtime environment	CRITICAL

Escalate to CRITICAL and activate IC if:

• The compromised package is confirmed to contain intentionally injected malicious code (not just a bug)
• The package executed during a production build or runtime in the window when the malicious version was active
• Any unexpected outbound network connections are observed from Arbitex services

---

3. Immediate actions (first 15 minutes)

Step	Action	Owner
1	Declare incident	IC
2	Identify all Arbitex repos that use the affected package	Engineering
3	Determine whether the malicious version is in production	Engineering
4	Pin all affected repos to last known-good version immediately	Engineering
5	Halt any running builds or deployments using the affected package	Engineering
6	Freeze audit log if runtime compromise is suspected	IC

Step 1 — Declare incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/declare" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Supply chain attack — compromised dependency: <package_name>@<version>",
    "severity": "CRITICAL",
    "playbook": "IR-11"
  }' | jq '{incident_id: .id, ic_expires: .ic_window_expires}'

Step 2 — Identify affected repos:

Check each Arbitex repo for the compromised package. The repos and their package managers are:

Repo	Path	Package manager
platform (backend)	$ARBITEX_ROOT/git/arbitex-platform/backend/	Python (pyproject.toml / requirements.txt)
platform (frontend)	$ARBITEX_ROOT/git/arbitex-platform/frontend/	npm (package-lock.json)
cloud (backend)	$ARBITEX_ROOT/git/arbitex-cloud/backend/	Python
cloud (frontend)	$ARBITEX_ROOT/git/arbitex-cloud/frontend/	npm
outpost	$ARBITEX_ROOT/git/arbitex-outpost/	Go (go.mod / go.sum)
core	$ARBITEX_ROOT/git/arbitex-core/	Python
www	$ARBITEX_ROOT/git/arbitex-www/	npm
docs	$ARBITEX_ROOT/git/arbitex-docs/	npm

# Check for a specific npm package across all repos with package-lock.json
grep -r '"<package_name>"' arbitex-*/frontend/package-lock.json \
  arbitex-www/package-lock.json \
  arbitex-docs/package-lock.json 2>/dev/null \
  | grep '"version"'

# Check for a Python package
grep -r '<package_name>' arbitex-*/backend/requirements.txt \
  arbitex-core/pyproject.toml 2>/dev/null

Step 3 — Determine whether malicious version is in production:

Check git history to find when the compromised version was introduced:

# In the affected repo — find when the package was added/updated
git -C <repo_path> log --all --follow -p -- <lockfile_path> \
  | grep -A2 '"<package_name>"' | head -40

Compare the introduction date against the reported malicious version publication date. If the malicious version was deployed to production before detection, proceed to investigation section 4 immediately.

Step 4 — Pin to last known-good version:

Do NOT run npm install or pip install without specifying an exact known-good version. Use lockfile-only installs until the incident is resolved.

# npm — pin to last known-good version in package.json, then install from lockfile only
npm install --package-lock-only <package_name>@<last_good_version>
npm ci  # install ONLY from the lockfile — does not update lockfile

# pip — pin exact version in requirements.txt
# Edit requirements.txt: change <package_name>>=<bad_version> to <package_name>==<last_good_version>
pip install -r requirements.txt --require-hashes

Step 5 — Halt active builds:

If a CI/CD pipeline is actively building with the compromised version, cancel the build before it produces an image that could be deployed. Contact the engineering team immediately — do not wait for builds to finish.

Step 6 — Freeze audit log (if runtime compromise suspected):

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/audit/freeze" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-11: Supply chain attack — preserving audit evidence, runtime compromise suspected"
  }'

---

4. Investigation

Determine whether the malicious payload executed, and if so, what it did.

Compare compromised and known-good package versions:

Retrieve the package contents from a secure mirror or from the package registry cache (before it was pulled). Compare file hashes between the compromised and clean versions:

# npm — compare package contents
npm pack <package_name>@<last_good_version> -o /tmp/good-pkg.tgz
npm pack <package_name>@<compromised_version> -o /tmp/bad-pkg.tgz
diff <(tar -tzf /tmp/good-pkg.tgz | sort) <(tar -tzf /tmp/bad-pkg.tgz | sort)

Look for: new files (especially scripts), modified package.json scripts (install hooks, postinstall), changes to source files that touch network I/O or file system access.

Review git history for when the compromised version was introduced:

git -C <repo_path> log --oneline --since=<package_registry_compromise_date> -- <lockfile_path>

Check container image build timestamps:

Container images are built via CI and stored in Azure Container Registry (ACR). Identify images built during the window when the compromised version was active:

# List recent images in ACR (requires az CLI with appropriate access)
az acr repository show-tags --name abtxacr --repository arbitex/<service_name> \
  --orderby time_desc --output table | head -20

Any image built after the compromised version was introduced and before the pin is treated as potentially tainted.

Check for indicators of compromise in runtime environment:

If the malicious package was active in a production runtime:

# Check for unexpected outbound connections (from platform or staff service hosts)
# This requires access to cloud network flow logs — coordinate with infrastructure team

# Review audit events for unusual admin API calls during the compromise window
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?created_after=<compromise_start_iso>&action=admin.config_read&limit=500" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq '.events[] | {timestamp, action, source_ip, user_id}'

Verify HMAC audit chain integrity for the compromise window:

curl -s -X POST "https://api.arbitex.ai/api/v1/admin/audit/verify" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  | jq '{valid, total_entries, errors}'

A valid: false result during the compromise window — combined with confirmed malicious package execution — should be treated as evidence of active audit tampering.

---

5. Containment

Scale containment based on confirmed scope.

Scope	Action
Malicious version never ran in production	Pin package, rebuild images — no emergency controls needed
Malicious version ran in production but no IOC found	Rebuild and redeploy all services, rotate all secrets as precaution
Confirmed IOC — unexpected network connections	Rotate ALL secrets immediately, activate emergency kill switch
Confirmed data exfiltration	Full secret rotation, tenant isolation for affected orgs, escalate to legal

Rebuild all container images from a clean base after pinning:

# After pinning the affected package to the known-good version:
# 1. Commit the updated lockfile to git
# 2. Trigger a fresh CI build — the build pipeline will produce new images
# 3. Confirm the new image build does NOT use the compromised package version
npm ci --audit  # verify clean after rebuild in Node repos

Rotate all secrets if runtime compromise is suspected:

If the malicious payload could have read environment variables or files at runtime, all secrets must be rotated. The critical secrets for each service are:

Service	Secrets to rotate
Platform	Database credentials, Redis password, HMAC signing key, JWT signing keys, API keys, Key Vault access credentials
Cloud	Database credentials, session secret, Stripe API keys, Key Vault access
Outpost	mTLS client certificate, credential sync token, policy bundle signing key
Staff	FIDO/WebAuthn RP credentials, database credentials, session secrets

Rotate platform-level credentials via the emergency endpoint:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/credentials/revoke-all" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "scope": "platform",
    "reason": "IR-11: Supply chain attack — runtime compromise suspected, rotating all credentials as precaution"
  }' | jq '{revoked_count: .revoked}'

Database, Redis, and Key Vault credentials must be rotated through Azure portal and operations tooling — they are not managed through the platform API.

Activate emergency kill switch if active exfiltration is observed:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/killswitch" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-11: Supply chain attack — active exfiltration suspected, halting all provider traffic"
  }'

---

6. Recovery

Recovery requires a verified clean build and confirmed secret rotation. Do not re-enable traffic until all steps below are complete.

Step	Action	Verification
1	Pin affected package to patched version (or safe alternative)	npm audit --omit=dev exits 0
2	Rebuild all affected container images from clean base	Review build logs — confirm no compromised version present
3	Redeploy all services with clean images	Health checks pass post-deploy
4	Run full test suite post-redeploy	pytest exit code 0, all CI checks green
5	Rotate all secrets if runtime compromise occurred	New credentials active, old credentials revoked
6	Verify SBOM is clean	Generate SBOM from new build artifacts
7	Verify platform health	GET /readyz, GET /health/deep
8	Close incident	POST /api/staff/incident/{id}/close

Platform health verification post-redeploy:

curl -s "https://api.arbitex.ai/readyz" | jq '.'
curl -s "https://api.arbitex.ai/health/deep" | jq '{status, db, redis, providers, credint}'

Close the incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/close" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Compromised package pinned, images rebuilt, services redeployed, secrets rotated, SBOM verified clean",
    "root_cause": "Compromised upstream package <package_name>@<version> — <source: typosquatting/maintainer compromise/advisory>",
    "data_exfiltrated": false
  }'

---

7. Communication

Internal notification to all engineering teams (immediate)

Send when the affected package and scope are confirmed:

Subject: [IR-11 ACTIVE] Supply chain compromise — <package_name>@<version>

Incident ID: <incident_id>
Declared: <timestamp>
Severity: CRITICAL

Affected package: <package_name>@<version>
Malicious version published: <datetime>
Affected Arbitex repos: <list>

Status: Package pinned to <last_good_version>. Rebuilding affected images.
Runtime execution confirmed: [YES / NO / UNKNOWN]

Action required from all engineers:
- Do NOT run npm install / pip install in affected repos until cleared
- Do NOT deploy any image built between <compromise_start> and <pin_datetime>
- Report any anomalous behavior observed in the past <N> days to IC immediately

IC: <name>
Next update: <30 minutes from now>

Customer notification (only if data exfiltration is confirmed)

If investigation confirms the malicious payload exfiltrated customer data, follow IR-5 escalation procedures and notify affected customers per the data breach notification process. Provide:

• Timeline of when the compromised package was active in production
• Nature of data potentially exposed (determined by what the payload accessed)
• Steps Arbitex has taken to contain and remediate

Compliance records

Document the following for compliance records regardless of outcome:

• Package name, compromised version, last known-good version
• Date compromised version was introduced to Arbitex repos (from git history)
• Date detected and response actions taken
• Whether the malicious version executed in a production environment
• SBOM snapshot before and after remediation

---

8. Post-incident

Evidence preservation checklist:

Item	Status
Compromised package version archived (tarball) for forensic analysis	[ ]
Git history showing when compromised version was introduced, recorded	[ ]
List of container image tags built during the compromise window, archived	[ ]
Audit export (compromise window) saved to secure storage	[ ]
HMAC chain verification result recorded	[ ]
Network flow logs for the compromise window requested from infrastructure team	[ ]
SBOM from pre-compromise and post-remediation builds archived	[ ]
Incident timeline document created	[ ]

Lessons learned template:

• Which automated control detected this — Dependabot, npm audit, manual audit, or external advisory? How long was the package in production before detection?
• Was there a postinstall or build-time hook that executed the malicious code? Could build sandboxing have prevented execution?
• Were all affected repos identified within the first 15 minutes? If not, what slowed the cross-repo search?
• Did any secrets need to be rotated? If so, was the rotation process fast enough to limit exposure?
• Should the sprint-close dependency audit cadence be increased to weekly automated scanning?

Related playbooks:

• If secrets were confirmed exfiltrated and used: follow IR-1: Compromised Tenant Account for affected orgs
• If HMAC chain shows tampering: follow IR-7: HMAC Chain Break Detected