Skip to content
Arbitex Docs

IR-5: Database Breach Suspected

Severity: CRITICAL

Scope: Suspected unauthorized access to the Arbitex PostgreSQL database — anomalous queries, unauthorized connections, data appearing outside the platform, or compromised database credentials.

IC role required for: audit freeze, emergency credential revocation, tenant isolation.

Related documentation: Audit Log Verification — Audit Chain Integrity — Credential Management — Security Operations

1. Detection

Section titled “1. Detection”

A database breach may be detected through infrastructure monitoring, application audit events, or external reporting. All of the following should be treated as high-confidence indicators.

Signal Source Details
Unusual query patterns in PostgreSQL logs pg_log / Azure Database audit Unexpected COPY, SELECT * on sensitive tables, or pg_dump-style bulk reads
Connection attempts from unknown IPs PgBouncer connection logs IPs not in the expected application pod CIDR or admin bastion range
PgBouncer pool anomalies PgBouncer stats Pool saturation, unexpected client count, unknown database names being connected to
Unexpected active sessions in pg_stat_activity Direct database query Sessions from unknown application_name values or unexpected user roles
Data found outside the platform Customer report or threat intelligence Platform data appearing in pastebin, dark web, or competitor systems
Application alerts: unexpected admin operations Audit log Bulk exports, unusual config reads, or sensitive table scans via the application layer
Database credential seen in external breach data CredInt alert or external notification Database password hash or connection string found in known breach corpus

Any single confirmed indicator is sufficient to declare this incident. Do not wait for multiple signals before activating.

2. Severity classification

Section titled “2. Severity classification”
Classification Criteria
CRITICAL (default) Confirmed unauthorized connection to the database, or data confirmed outside the platform
HIGH Anomalous query patterns or connection attempts, breach not yet confirmed
HIGH Database credentials found in breach corpus — treat as confirmed until rotated and verified

Downgrade from CRITICAL only after investigation confirms no unauthorized access occurred and anomalies have an innocent explanation.

3. Immediate actions (first 15 minutes)

Section titled “3. Immediate actions (first 15 minutes)”

Execute steps 1 through 5 in order without waiting for investigation results. Speed of credential rotation limits the attacker’s window.

Step Action Done
1 Declare the incident
2 Freeze the audit log
3 Take an immediate database snapshot for forensics
4 Enable enhanced PostgreSQL logging
5 Rotate the database credentials

Step 1 — Declare the incident:

Terminal window
curl -X POST https://api.arbitex.ai/api/staff/incident/declare \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "IR-5: Database breach suspected",
    "severity": "critical",
    "playbook": "ir-5-database-breach"
  }'

Note the incident_id from the response.

Step 2 — Freeze the audit log immediately:

Terminal window
curl -X POST https://api.arbitex.ai/api/staff/emergency/audit/freeze \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "Suspected database breach — preserving audit trail before any writes"
  }'

Freezing is critical: if the attacker has write access to the database and the audit table, they may attempt to delete or modify audit records. Freezing the audit log via the API locks the write path at the application layer. The HMAC chain provides independent tamper evidence — verify it as part of the investigation.

Step 3 — Create forensic database snapshot:

Terminal window
# Azure CLI — create a point-in-time restore point before any remediation actions
az postgres flexible-server restore \
  --resource-group arbitex-prod-rg \
  --name arbitex-db-forensic-<incident_id> \
  --source-server arbitex-db \
  --restore-point-in-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

This snapshot is read-only forensic evidence. Do not use it as the production database — its purpose is to preserve the state at the moment of detection.

Step 4 — Enable enhanced PostgreSQL logging on the production database:

Terminal window
# Enable statement logging for forensic purposes
az postgres flexible-server parameter set \
  --resource-group arbitex-prod-rg \
  --server-name arbitex-db \
  --name log_statement \
  --value all


az postgres flexible-server parameter set \
  --resource-group arbitex-prod-rg \
  --server-name arbitex-db \
  --name log_connections \
  --value on


az postgres flexible-server parameter set \
  --resource-group arbitex-prod-rg \
  --server-name arbitex-db \
  --name log_disconnections \
  --value on

Note: log_statement = all will increase log volume significantly. Revert after the investigation window (see Recovery).

Step 5 — Rotate database credentials:

Database credential rotation is an infrastructure operation performed directly — not through the Arbitex admin API. Coordinate with the infrastructure team to:

  1. Generate new credentials for all database users: arbitex_app, arbitex_readonly, arbitex_migration.
  2. Update the credential in Azure Key Vault.
  3. Rolling-restart the platform pods to pick up the new credential via the secrets backend.
  4. Update the PgBouncer userlist.txt with the new password hash.
  5. Verify PgBouncer pool reconnects successfully after the restart.
Terminal window
# After rotation — verify platform pods are reconnected
kubectl -n arbitex get pods -l app=arbitex-platform


# Verify database connectivity via readiness probe
curl https://api.arbitex.ai/readyz | jq '.checks.database'

The /readyz endpoint reports database connectivity status. A "status": "ok" from the database check confirms the platform has reconnected with the new credentials.

4. Investigation

Section titled “4. Investigation”

Active session review

Section titled “Active session review”

Immediately after credential rotation, check for any sessions that survived the credential change. Unexpected surviving sessions indicate the attacker has a separate authentication path (a new role they created, a stored procedure with security definer, or a connection not routed through PgBouncer).

-- Run against the production database (read-only access via bastion)
SELECT
  pid,
  usename,
  application_name,
  client_addr,
  state,
  query_start,
  left(query, 200) AS current_query
FROM pg_stat_activity
WHERE usename NOT IN ('arbitex_app', 'arbitex_readonly', 'arbitex_migration', 'rdsadmin', 'azure_pg_admin')
   OR client_addr NOT IN (/* expected application CIDR */);

Terminate any unexpected sessions:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid = <unexpected_pid>;

PgBouncer log review

Section titled “PgBouncer log review”

Review PgBouncer connection logs for the suspected breach window:

Terminal window
kubectl -n arbitex logs deployment/pgbouncer --since=720h | \
  grep -v "arbitex_app\|arbitex_readonly" | \
  grep "LOGIN\|CONNECT\|AUTH"

Look for:

  • Connections from IPs outside the application pod CIDR
  • Authentication attempts for database usernames that do not exist in the schema
  • Login failures followed by successful logins (credential stuffing)
  • Connections to database names other than arbitex

Table access analysis

Section titled “Table access analysis”

Query the PostgreSQL audit log (via Azure Database for PostgreSQL audit log export) to identify which tables were accessed:

Terminal window
# Azure Monitor log query (KQL) — tables accessed in the breach window
AzureDiagnostics
| where ResourceType == "SERVERS" and Category == "PostgreSQLLogs"
| where TimeGenerated between (<start_time> .. <end_time>)
| where Message contains "SELECT" or Message contains "COPY"
| extend table_name = extract("FROM (\\w+)", 1, Message)
| summarize count() by table_name, CallerIPAddress
| order by count_ desc

Tables of highest concern in order of sensitivity:

Table Data Risk
audit_events Full audit trail Evidence destruction
users Account credentials (hashed), PII Account compromise
api_keys Key hashes, metadata Lateral movement
credentials Secret hashes, key prefixes API access
conversations User prompt/response content Data exfiltration
dlp_scan_results Redacted PII classifications Privacy exposure
org_config Tenant configuration, policy rules Configuration theft

Application audit trail

Section titled “Application audit trail”

Export audit events for the investigation window via the application layer:

Terminal window
curl -X POST https://api.arbitex.ai/api/v1/admin/audit/export \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "start_date": "<breach_window_start_date>",
    "end_date": "<incident_detection_date>",
    "action": "admin.config_read",
    "format": "jsonl",
    "include_hmac": true
  }'

HMAC chain integrity verification

Section titled “HMAC chain integrity verification”

Verify the audit chain for the breach window to confirm no audit entries were deleted or modified:

Terminal window
curl -X POST https://api.arbitex.ai/api/v1/admin/audit/verify \
  -H "Authorization: Bearer $ADMIN_TOKEN"

If valid: false is returned, the errors array will identify the first broken link. A break in the HMAC chain indicates one or more audit entries were deleted or modified after initial write. The HMAC-SHA256 chain uses a genesis sentinel of 64 zero hex characters — any deviation from the chain at any point is evidence of tampering.

5. Containment

Section titled “5. Containment”
Control Action
Rotate database credentials Complete — done in Immediate Actions
Restrict database network access Lock PostgreSQL security group/firewall to application pod CIDR only
Require SSL for all connections Verify ssl_require is set and no sslmode=disable connections are accepted
Review and drop unexpected database roles Check pg_roles for any roles not in the expected set
Freeze audit log Complete — done in Immediate Actions
Check for unauthorized stored procedures or triggers Review pg_proc for newly added functions

Restrict database network access:

Terminal window
# Lock the PostgreSQL flexible server firewall to known application IPs only
az postgres flexible-server firewall-rule delete \
  --resource-group arbitex-prod-rg \
  --name arbitex-db \
  --rule-name AllowAll


# Add explicit rule for application pod IP range only
az postgres flexible-server firewall-rule create \
  --resource-group arbitex-prod-rg \
  --name arbitex-db \
  --rule-name ArbitexAppPods \
  --start-ip-address <app_pod_cidr_start> \
  --end-ip-address <app_pod_cidr_end>

Verify no unexpected roles exist:

SELECT rolname, rolsuper, rolcreaterole, rolcreatedb, rolcanlogin
FROM pg_roles
WHERE rolname NOT IN (
  'arbitex_app', 'arbitex_readonly', 'arbitex_migration',
  'azure_pg_admin', 'rdsadmin', 'postgres'
)
ORDER BY rolname;

Any result from this query represents a role that was not provisioned by the Arbitex deployment process. Treat as a backdoor and drop immediately after documenting for forensics.

6. Recovery

Section titled “6. Recovery”

Do not unfreeze the audit log or restore normal database access until all verification steps pass.

Step Verification
Confirm no unexpected roles remain pg_roles query returns empty
Confirm no unexpected stored procedures pg_proc review finds no unauthorized functions
Confirm pg_hba.conf has not been modified Hash of pg_hba.conf matches deployment-time value
Confirm no unauthorized extensions installed SELECT * FROM pg_extension shows only expected extensions
Confirm new credentials accepted by all platform pods /readyz database check returns ok
Verify HMAC chain intact for post-rotation audit entries POST /api/v1/admin/audit/verify (verifies caller’s org)
Revert enhanced logging Set log_statement = none after investigation window closes
Unfreeze audit log After all evidence exports confirmed complete

Revert enhanced logging:

Terminal window
az postgres flexible-server parameter set \
  --resource-group arbitex-prod-rg \
  --server-name arbitex-db \
  --name log_statement \
  --value none

Close the incident:

Terminal window
curl -X POST https://api.arbitex.ai/api/staff/incident/<incident_id>/close \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Database credentials rotated. No unauthorized roles or backdoors found. HMAC chain intact. Network access restricted to application pod CIDR."
  }'

7. Communication

Section titled “7. Communication”

Internal notification (immediate — within 15 minutes)

Section titled “Internal notification (immediate — within 15 minutes)”

Send to: Engineering lead, Security lead, Legal, Customer Success lead.

Subject: [IR-5 ACTIVE] Database breach suspected


Incident ID: <incident_id>
Declared: <timestamp>
Severity: CRITICAL


Summary: Suspected unauthorized access to the Arbitex PostgreSQL database
detected at <detection_time>. Immediate containment actions taken:
- Audit log frozen at <timestamp>
- Forensic snapshot created: arbitex-db-forensic-<incident_id>
- Database credential rotation in progress


Current status: Investigating which tables were accessed and whether data
was exfiltrated.


Next update in 30 minutes.
IC: <name>
Section titled “Legal notification (within 1 hour)”

Send to: General Counsel, DPO (if applicable).

Subject: [LEGAL HOLD] Potential data breach — IR-5 <incident_id>


A potential database breach is under active investigation. Please note:


1. A legal hold is in effect for all evidence related to incident <incident_id>
2. The forensic database snapshot arbitex-db-forensic-<incident_id> must not
   be deleted pending investigation
3. This may trigger breach notification obligations under GDPR, CCPA, or
   other applicable regulations depending on investigation findings
4. We will provide a preliminary data exposure assessment within 4 hours


Please advise on notification timelines and obligations for the jurisdictions
of our affected customers.

Customer notification (after scope is confirmed)

Section titled “Customer notification (after scope is confirmed)”

Send only after investigation determines which tenants were affected.

Subject: Important security notification — Arbitex platform


We are writing to inform you of a security incident that may have affected
your organization's data on the Arbitex platform.


Incident reference: <incident_id>
Detection time: <detection_timestamp UTC>
Containment time: <containment_timestamp UTC>


What happened: We detected indicators of unauthorized access to the Arbitex
database during the period <start_time> to <end_time UTC>.


What data was potentially affected: [Specify based on investigation —
e.g., "Conversation content stored in your organization's workspace" or
"User account metadata for your organization's users"]


What we have done: [Specific containment steps taken]


What you should do: [Specific customer actions, if any — e.g., rotate API keys,
notify your own users]


We will provide a full incident report within 5 business days.


Arbitex Security Team
incident-<incident_id>@arbitex.ai

8. Post-incident

Section titled “8. Post-incident”
Task Owner Timing
Archive forensic snapshot and audit export with HMAC fields Security Before closing incident
Root cause analysis — how was the database accessed? Engineering + Security Within 72 hours
Full table access report by tenant — what data was accessible? Engineering Within 48 hours
Regulatory breach notification assessment Legal + DPO Within 72 hours of confirmation
Implement additional database access controls identified during investigation Engineering Within 1 sprint
Customer root cause report Security Within 5 business days
Review and update database hardening documentation Docs team Within 1 sprint

For regulatory breach notification assessment, the key data points Legal will need:

  • Number of tenants affected
  • Categories of personal data potentially accessed (names, email addresses, conversation content, etc.)
  • Number of data subjects affected (estimated)
  • Time window of potential access
  • Whether the attacker appears to have exported or copied data vs. only querying it
  • Whether the HMAC chain integrity was compromised (indicating potential audit trail modification)