IR-8: Redis Compromise

A Redis compromise means an unauthorized actor has gained read or write access to the Arbitex Redis instance. Redis holds active sessions, the JWT blacklist, rate limiting counters, and application cache. A compromise may allow an attacker to read live session tokens, remove JWT blacklist entries to re-validate revoked tokens, or manipulate rate limit state to enable volumetric abuse.

For background on Redis usage patterns in the platform, see Security Architecture. For session management procedures, see Security Operations.

---

1. Detection

Redis compromise surfaces through operational anomalies rather than direct alerts, since Redis typically lacks application-layer audit logging.

Signal	Source	Notes
Redis SLOWLOG shows unexpected commands	redis-cli SLOWLOG GET	Look for KEYS *, DEBUG, CONFIG SET, SLAVEOF
Redis connections from unexpected IPs	Redis CLIENT LIST, network logs	Application pods should be the only clients
Redis AUTH failures in platform logs	Platform log aggregator	Credential stuffing against Redis port
Unusual memory growth	Prometheus redis_memory_used_bytes	Unexpected data written to keyspace
Rate limiting not enforced	Customers bypassing configured rate limits	Counter keys may have been deleted or reset
Blacklisted JWT accepted	jwt.blacklist_miss audit event	Blacklist key deleted from Redis
MONITOR output shows unexpected patterns	Redis MONITOR command (use briefly)	Direct access evidence

Investigate current connections and slow log:

# Check active Redis connections
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  CLIENT LIST | grep -v "addr=127.0.0.1"

# Review slow log for suspicious commands
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  SLOWLOG GET 100 | grep -E "(KEYS|CONFIG|DEBUG|SLAVEOF|REPLICAOF|BGSAVE|BGREWRITEAOF)"

# Check if any CONFIG changes were made
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  CONFIG GET save
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  CONFIG GET requirepass

---

2. Severity classification

Condition	Severity
Suspicious connection or command observed, no data access confirmed	MEDIUM
Unauthorized read access confirmed — session tokens may have been extracted	HIGH
JWT blacklist entries confirmed deleted or overwritten	HIGH
Session tokens confirmed extracted and used for unauthorized API access	CRITICAL
Redis used to bypass authentication (blacklist cleared)	CRITICAL

Escalate to CRITICAL immediately if:

• Audit logs show API requests authenticated with tokens that should have been blacklisted
• Session tokens observed in network captures or Redis exports originating outside platform pods
• Any indication an attacker is using Redis access to pivot to application-layer access

---

3. Immediate actions (first 15 minutes)

Warning: The primary containment action — FLUSHALL followed by credential rotation — invalidates all active user sessions platform-wide. Every user will be logged out and must re-authenticate. This is intentional and necessary. Notify internal stakeholders before executing if possible, but do not delay if active exploitation is confirmed.

Step	Action	Command
1	Declare incident	POST /api/staff/incident/declare
2	Freeze audit log	POST /api/staff/emergency/audit/freeze
3	Capture Redis state for forensics	redis-cli DEBUG JMAP / key scan
4	Rotate Redis password	Update REDIS_PASSWORD secret in Key Vault
5	Flush all Redis data	redis-cli FLUSHALL
6	Restart platform instances	Rolling restart to pick up new credentials

Step 1 — Declare incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/declare" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Redis compromise — unauthorized access suspected",
    "severity": "MEDIUM",
    "playbook": "IR-8",
    "notes": "Suspicious Redis activity detected. Preparing for FLUSHALL + credential rotation."
  }' | jq '{incident_id: .id, ic_expires: .ic_window_expires}'

Step 2 — Freeze audit log:

curl -s -X POST "https://api.arbitex.ai/api/staff/emergency/audit/freeze" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "<incident_id>",
    "reason": "IR-8: Redis compromise — preserving audit evidence"
  }' | jq '{frozen: .frozen, freeze_expires: .expires_at}'

Step 3 — Capture Redis key inventory for forensics:

Before flushing, capture as much state as possible. This preserves evidence of what an attacker could have read:

# Capture all top-level key patterns and counts (do not dump raw session tokens)
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  --scan --pattern '*' | sed 's/:.*//' | sort | uniq -c | sort -rn \
  > redis-key-inventory-$(date +%Y%m%dT%H%M%S).txt

# Capture connection history from slow log
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls \
  SLOWLOG GET 1000 > redis-slowlog-$(date +%Y%m%dT%H%M%S).txt

# Capture Redis info stats
redis-cli -h <redis_host> -a $REDIS_PASSWORD --tls INFO all \
  > redis-info-$(date +%Y%m%dT%H%M%S).txt

Step 4 — Rotate Redis password:

Rotate the REDIS_PASSWORD secret in Azure Key Vault. The application reads this at startup — all instances must restart after rotation.

# Generate a new 64-character random password
NEW_REDIS_PASS=$(openssl rand -hex 32)

# Update in Azure Key Vault
az keyvault secret set \
  --vault-name <vault_name> \
  --name "redis-password" \
  --value "$NEW_REDIS_PASS"

# Update in Redis itself (requires current credentials)
redis-cli -h <redis_host> -a $OLD_REDIS_PASSWORD --tls \
  CONFIG SET requirepass "$NEW_REDIS_PASS"

Step 5 — Flush all Redis data:

redis-cli -h <redis_host> -a $NEW_REDIS_PASS --tls FLUSHALL

FLUSHALL removes all data from all databases. This is a destructive operation with the following intended consequences:

• All active sessions are invalidated — all users must re-authenticate
• JWT blacklist entries are removed — mitigated by short token TTL (tokens expire naturally within minutes)
• Rate limiting counters reset — acceptable transient state
• Application cache cleared — cold start on next requests

Step 6 — Rolling restart of all platform instances:

Restart all platform pods to pick up the new REDIS_PASSWORD from Key Vault:

# In Kubernetes
kubectl rollout restart deployment/arbitex-platform -n arbitex
kubectl rollout status deployment/arbitex-platform -n arbitex

After restart, verify Redis connectivity:

curl -s "https://api.arbitex.ai/readyz" | jq '.redis'

---

4. Investigation

Conduct investigation after immediate containment is complete. The forensic window is limited because FLUSHALL was required for containment.

Check for unauthorized Redis access via network logs:

# Review platform network policy logs for connections to Redis port
# In Calico
kubectl logs -n calico-system -l app=calico-node | grep "<redis_ip>:6379"

# In cloud network security groups / flow logs
az network watcher flow-log list --resource-group <rg> --location <region>

Review Redis slow log captured in Step 3 for evidence of what the attacker did:

Key patterns that indicate active exploitation:

Command pattern	Implication
KEYS session:*	Session token enumeration
GET session:<id>	Session token extraction
DEL jwt_blacklist:*	JWT blacklist clearing
SET rate_limit:* with 0	Rate limit bypass
DEBUG RELOAD	Potentially used to disable persistence checks
CONFIG SET dir + CONFIG SET dbfilename	RDB export to attacker-controlled path
SLAVEOF <external_ip>	Replication to external host (data exfil)

Check whether blacklisted tokens were used:

# Look for authentication events using tokens issued to previously-revoked sessions
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=jwt.blacklist_miss&created_after=<detection_time_iso>" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq '.events'

If jwt.blacklist_miss events exist after the Redis compromise, the attacker actively exploited the cleared blacklist. Treat the associated sessions as fully compromised — follow IR-1: Compromised Tenant Account for each affected org.

Check for RDB/AOF file exfiltration:

# Verify Redis persistence configuration was not changed
redis-cli -h <redis_host> -a $NEW_REDIS_PASS --tls CONFIG GET save
redis-cli -h <redis_host> -a $NEW_REDIS_PASS --tls CONFIG GET dir
redis-cli -h <redis_host> -a $NEW_REDIS_PASS --tls CONFIG GET dbfilename

If dir was changed to an unusual path or save enabled unexpected snapshots, the attacker may have exported an RDB snapshot.

---

5. Containment

After the immediate actions in Section 3 are complete, harden the Redis network posture to prevent recurrence.

Action	Purpose	How
Restrict Redis to application pod CIDR only	Prevent future unauthorized direct access	Kubernetes NetworkPolicy or Calico
Verify TLS is enforced on Redis connections	Prevent credential sniffing	redis-cli CONFIG GET tls-port
Disable Redis MONITOR command	Prevent passive traffic interception	redis-cli CONFIG SET monitor-slow-log-max-len 0
Disable dangerous Redis admin commands	Prevent CONFIG, DEBUG abuse	Redis ACLs
Review Redis access control list	Ensure only platform service accounts have access	redis-cli ACL LIST

Apply Kubernetes NetworkPolicy to restrict Redis access:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: redis-restrict
  namespace: arbitex
spec:
  podSelector:
    matchLabels:
      app: redis
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: arbitex-platform
    ports:
    - protocol: TCP
      port: 6379

Set Redis ACLs to disable dangerous commands:

redis-cli -h <redis_host> -a $NEW_REDIS_PASS --tls \
  ACL SETUSER platform-svc on >$PLATFORM_REDIS_PASS \
  ~* &* +@all -DEBUG -MONITOR -SLAVEOF -REPLICAOF -CONFIG -BGSAVE -BGREWRITEAOF

---

6. Recovery

Recovery proceeds in parallel with containment hardening.

Step	Action	Verification
1	Verify sessions are re-established	Users can log in and get new sessions
2	Verify rate limiting is re-active	Check Prometheus rate_limit_hits_total counter
3	Verify JWT blacklist is repopulating	jwt_blacklist_size metric > 0 after logins
4	Verify Redis TLS	redis-cli --tls INFO server shows tls_enabled:1
5	Monitor for anomalous sessions	Section 4 blacklist miss check, repeat hourly

Check JWT blacklist repopulation:

After FLUSHALL, previously-blacklisted JWTs are absent from Redis. The blacklist repopulates as revocation events are re-processed. For tokens still within their TTL window that should have been blacklisted:

# Check for any revocation events in audit log that need to be replayed
curl -s "https://api.arbitex.ai/api/v1/admin/audit-logs/?action=credential.revoked&created_after=<72h_ago_iso>" \
  -H "Authorization: Bearer $STAFF_TOKEN" | jq '[.events[] | {id, user_id, revoked_at}]'

For any credential revoked in the last token TTL window (typically 15 minutes), manually re-revoke to force re-entry into the blacklist:

curl -s -X DELETE "https://api.arbitex.ai/api/v1/admin/credentials/<credential_id>" \
  -H "Authorization: Bearer $STAFF_TOKEN"

Verify platform health:

curl -s "https://api.arbitex.ai/readyz" | jq '.'
curl -s "https://api.arbitex.ai/health/deep" | jq '{status, db, redis, providers}'

---

7. Communication

Internal notification (within 30 minutes — before FLUSHALL)

Notify platform engineering and on-call team before executing the FLUSHALL, as it will trigger a platform-wide session invalidation:

Subject: [IR-8] Redis compromise — platform-wide session invalidation in ~5 minutes

Incident ID: [INCIDENT_ID]
Detected: [ISO_TIMESTAMP]
Action: We are executing FLUSHALL + Redis credential rotation due to a suspected Redis compromise.

Impact: ALL active user sessions will be invalidated. Every authenticated user will be logged out and must re-authenticate. This includes staff sessions.

Expected disruption window: ~[N] minutes during rolling platform restart.

IC: [IC_NAME] — [CONTACT]

Customer-facing status page update (if session disruption is visible)

[STATUS_PAGE_TITLE]: Planned Security Maintenance — Re-authentication Required

We performed a security maintenance operation that has invalidated all active sessions.
All users will be required to log in again. This is expected behavior.

Duration: [START] – [END]
Impact: Session invalidation only — no data loss. All API keys and configurations are unaffected.

Escalation (if session tokens confirmed extracted and used)

If investigation confirms that session tokens were extracted and used by an unauthorized actor, escalate per IR-1: Compromised Tenant Account for each affected org. Provide:

• Incident ID
• List of affected org IDs
• Approximate session window during which tokens were accessible
• Whether any jwt.blacklist_miss events were observed

---

8. Post-incident

Evidence preservation checklist:

Item	Status
Redis key inventory snapshot saved	[ ]
Redis slow log captured before FLUSHALL	[ ]
Redis INFO all stats captured	[ ]
Network flow logs for Redis port captured	[ ]
JWT blacklist miss events queried and recorded	[ ]
jwt.blacklist_miss events investigated per org	[ ]
Redis ACL hardening applied	[ ]
NetworkPolicy restricting Redis access verified	[ ]

Close the incident:

curl -s -X POST "https://api.arbitex.ai/api/staff/incident/<incident_id>/close" \
  -H "Authorization: Bearer $STAFF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "resolution": "Redis flushed, credentials rotated, platform restarted. JWT blacklist integrity verified. Network access hardened.",
    "root_cause": "<unauthorized_network_access / credential_exposure / misconfiguration>",
    "data_exfiltrated": false
  }'

Lessons learned template:

• How did the attacker reach the Redis port? Was a NetworkPolicy in place?
• Was Redis TLS enforced? Was requirepass set?
• How long was the unauthorized access window before detection?
• Were any jwt.blacklist_miss events observed — meaning blacklisted tokens were actively replayed?
• Should Redis ACLs be applied to restrict dangerous commands in production?

Related playbooks:

• If blacklisted tokens were actively used post-compromise: follow IR-1: Compromised Tenant Account
• If Redis access was gained via a compromised application credential: follow IR-2: Compromised API Key