Splunk HEC SIEM integration deep dive

Arbitex forwards audit events to Splunk via the HTTP Event Collector (HEC) using the SplunkHECConnector class (backend/app/services/siem/splunk_hec.py). All events are emitted in OCSF v1.1 format with a hardcoded sourcetype of arbitex:ocsf. This guide covers everything you need to stand up and operate the integration end to end.

For a comparison of all seven Arbitex SIEM connectors, see the SIEM integration overview. For background on delivery path selection (Platform connector vs Outpost direct sink), see the SIEM integration guide.

---

Overview

The Splunk HEC connector operates server-side on the Arbitex Platform. When an audit event is generated — a DLP block, an authentication failure, an admin configuration change, or any other platform action — it is serialized to OCSF, placed in an in-memory buffer, and delivered to your Splunk HEC endpoint in batches.

Architecture flow:

Arbitex Platform (event source)
        |
        v
  [In-memory buffer]  <-- async background flush task
        |         \
        |          \-- flush every SPLUNK_HEC_FLUSH_INTERVAL seconds
        v
  [Batch (≤ SPLUNK_HEC_BATCH_SIZE events)]
        |
        v
  POST /services/collector  (Authorization: Splunk <token>)
        |
   -----+------
  |             |
200 OK       4xx/5xx
  |             |
Acked       Retry or dead letter

The connector batches up to SPLUNK_HEC_BATCH_SIZE events (default 100) and flushes on whichever comes first: the batch size limit or SPLUNK_HEC_FLUSH_INTERVAL seconds (default 5). A background asyncio task drives the flush loop, so event delivery is non-blocking relative to request processing.

Connector ID: splunk_hec

Source file: backend/app/services/siem/splunk_hec.py

Event format: OCSF v1.1.0, newline-delimited JSON HEC batch

Sourcetype: arbitex:ocsf (hardcoded — not configurable)

---

Prerequisites

Before configuring the Arbitex side, complete the following in your Splunk environment.

Splunk requirements:

• Splunk Enterprise 8.x+ or Splunk Cloud
• An HEC input enabled on port 8088 (or your custom port)
• An HEC token with write access to your target index
• A Splunk index to receive Arbitex events (recommended: arbitex)
• Network connectivity from your Arbitex Platform host to the Splunk HEC endpoint

Arbitex requirements:

• Platform version with SplunkHECConnector support
• Access to set environment variables (Kubernetes secret, Helm values, or .env file)
• Write access to the dead letter directory (default /var/log/arbitex/)

Tip

For Splunk Cloud, the HEC endpoint URL follows the pattern https://http-inputs-<stack>.splunkcloud.com:443/services/collector. Enable HEC in the Splunk Cloud self-service portal under Settings > Data Inputs > HTTP Event Collector.

---

Step-by-step setup

1. Create a Splunk index

   In Splunk Web, navigate to Settings > Indexes > New Index. Create an index named arbitex (or your preferred name). Set the retention policy appropriate for your compliance requirements.

   For Splunk Enterprise via CLI:

   splunk add index arbitex -datatype event -maxTotalDataSizeMB 50000

2. Create an HEC token

   Navigate to Settings > Data Inputs > HTTP Event Collector > New Token.

   • Name: arbitex-platform
   • Source type: Select Manual and enter arbitex:ocsf
   • Default index: arbitex
   • Allowed indexes: arbitex

   Copy the generated token — you will set this as SPLUNK_HEC_TOKEN.

3. Verify HEC endpoint accessibility

   From your Arbitex Platform host, confirm the HEC endpoint is reachable:

   curl -k -H "Authorization: Splunk <your-token>" \
     https://splunk.example.com:8088/services/collector \
     -d '{"event":"test"}'

   A response of {"text":"Success","code":0} confirms connectivity and token validity.

4. Set environment variables

   Configure the required variables on your Arbitex Platform deployment:

   Kubernetes secret

   apiVersion: v1
   kind: Secret
   metadata:
     name: arbitex-siem-splunk
     namespace: arbitex
   type: Opaque
   stringData:
     SPLUNK_HEC_URL: "https://splunk.example.com:8088/services/collector"
     SPLUNK_HEC_TOKEN: "your-hec-token-here"
     SPLUNK_HEC_INDEX: "arbitex"
     SPLUNK_HEC_SOURCE: "arbitex:audit"
     SPLUNK_HEC_BATCH_SIZE: "100"
     SPLUNK_HEC_FLUSH_INTERVAL: "5"
     SPLUNK_HEC_MAX_RETRIES: "3"
     SPLUNK_HEC_DEAD_LETTER_PATH: "/var/log/arbitex/splunk_dead_letter.jsonl"

   Helm values

   platform:
     env:
       SPLUNK_HEC_URL: "https://splunk.example.com:8088/services/collector"
       SPLUNK_HEC_TOKEN: "your-hec-token-here"
       SPLUNK_HEC_INDEX: "arbitex"
       SPLUNK_HEC_SOURCE: "arbitex:audit"

   Environment file

   SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector
   SPLUNK_HEC_TOKEN=ABA1FA15E000-example-hec-token
   SPLUNK_HEC_INDEX=arbitex
   SPLUNK_HEC_SOURCE=arbitex:audit
   SPLUNK_HEC_BATCH_SIZE=100
   SPLUNK_HEC_FLUSH_INTERVAL=5
   SPLUNK_HEC_MAX_RETRIES=3
   SPLUNK_HEC_DEAD_LETTER_PATH=/var/log/arbitex/splunk_dead_letter.jsonl

5. Verify the dead letter directory

   Ensure the directory exists and is writable by the Arbitex Platform process:

   mkdir -p /var/log/arbitex
   chown arbitex:arbitex /var/log/arbitex
   chmod 750 /var/log/arbitex

6. Restart the Arbitex Platform

   Apply the new environment variables by restarting the platform service:

   # Kubernetes
   kubectl rollout restart deployment/arbitex-platform -n arbitex
   
   # Docker Compose
   docker compose up -d --force-recreate platform

7. Confirm event delivery

   Generate a test event (e.g., trigger a DLP policy in the Cloud Portal) and verify it appears in Splunk:

   index=arbitex sourcetype="arbitex:ocsf" earliest=-5m
   | head 5

   You should see structured OCSF JSON events within 5–10 seconds of the triggering action.

---

Configuration reference

Environment variable	Required	Default	Description
SPLUNK_HEC_URL	Yes	—	Full HEC endpoint URL, e.g. https://splunk.example.com:8088/services/collector
SPLUNK_HEC_TOKEN	Yes	—	HEC authentication token created in Splunk
SPLUNK_HEC_INDEX	No	arbitex	Target Splunk index for all Arbitex events
SPLUNK_HEC_SOURCE	No	arbitex:audit	Value of the source field in the HEC envelope
SPLUNK_HEC_BATCH_SIZE	No	100	Maximum number of events per HTTP POST
SPLUNK_HEC_FLUSH_INTERVAL	No	5	Seconds between background flush cycles
SPLUNK_HEC_MAX_RETRIES	No	3	Maximum retry attempts for retriable errors
SPLUNK_HEC_DEAD_LETTER_PATH	No	/var/log/arbitex/splunk_dead_letter.jsonl	Path to the JSONL dead letter file

Caution

The sourcetype field is hardcoded to arbitex:ocsf in the connector source and cannot be changed via environment variable. Any sourcetype configured on your HEC token in Splunk will be overridden by the value in the event payload.

---

Sample configuration

Minimal configuration (required fields only)

SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector
SPLUNK_HEC_TOKEN=<YOUR_SIEM_TOKEN>

Production configuration

# Connection
SPLUNK_HEC_URL=https://splunk.corp.example.com:8088/services/collector
SPLUNK_HEC_TOKEN=<YOUR_SIEM_TOKEN>

# Index and metadata
SPLUNK_HEC_INDEX=arbitex
SPLUNK_HEC_SOURCE=arbitex:audit

# Delivery tuning
SPLUNK_HEC_BATCH_SIZE=100
SPLUNK_HEC_FLUSH_INTERVAL=5
SPLUNK_HEC_MAX_RETRIES=3

# Dead letter
SPLUNK_HEC_DEAD_LETTER_PATH=/var/log/arbitex/splunk_dead_letter.jsonl

HEC payload structure

The connector posts newline-delimited JSON to the HEC endpoint. Each request body contains one or more events separated by newlines (not a JSON array):

{"index":"arbitex","source":"arbitex:audit","sourcetype":"arbitex:ocsf","event":{"class_uid":2001,"severity_id":4,"actor":{"user":{"uid":"[email protected]","org_uid":"org_abc123"}},"finding_info":{"title":"Credential detected in response"},"raw_data":"credint_hit","time":1741824000123},"time":1741824000.123}
{"index":"arbitex","source":"arbitex:audit","sourcetype":"arbitex:ocsf","event":{"class_uid":6003,"actor":{"user":{"uid":"[email protected]"}},"unmapped":{"model_id":"gpt-4o","latency_ms":342,"provider":"openai"},"time":1741824001456},"time":1741824001.456}

The time field at the envelope level is a Unix epoch with millisecond precision (float). This ensures Splunk indexes events at the time they occurred on the platform, not the time HEC received them.

Dead letter file format

Events that exhaust all retry attempts (or fail with a non-retryable 4xx status) are written to the dead letter file as JSONL:

{"event":{"class_uid":2001,"severity_id":4,"actor":{"user":{"uid":"[email protected]"}}},"error":"HTTP 400 Bad Request: Invalid token","connector":"splunk_hec","timestamp":1741824005.0}
{"event":{"class_uid":6003,"unmapped":{"model_id":"gpt-4o"}},"error":"Max retries exceeded after 3 attempts","connector":"splunk_hec","timestamp":1741824010.0}

---

Retry and error handling

The connector uses exponential backoff for retriable errors. The delay before retry attempt n is min(2^n, 30) seconds:

Attempt	Delay
1 (initial)	—
2	2 seconds
3	4 seconds
4 (max)	8 seconds

Retriable errors: HTTP 429 (Too Many Requests) and HTTP 503 (Service Unavailable).

Non-retriable errors: Any 4xx response except 429 (e.g., 400 Bad Request, 401 Unauthorized, 403 Forbidden). These bypass the retry loop and go directly to the dead letter file.

Health check behavior: The connector performs a health check by POSTing an empty body to the HEC URL. Splunk returns HTTP 400 with "No data" when the endpoint is reachable but the payload is empty — the connector treats 200 and 400 both as HEALTHY. HTTP 403 indicates an invalid token and maps to ERROR. Any other response maps to DEGRADED.

Danger

A DEGRADED health status (any response other than 200, 400, or 403) means events may be accumulating in the buffer without delivery. Monitor the dead letter file size at SPLUNK_HEC_DEAD_LETTER_PATH and alert if it grows unexpectedly.

---

Example queries

The following SPL queries assume events are indexed at index=arbitex with sourcetype="arbitex:ocsf". Adjust the index name if you configured a different value for SPLUNK_HEC_INDEX.

DLP blocks by category and action

Summarizes DLP enforcement events with severity High or Critical (severity_id 4 and 5):

index=arbitex sourcetype="arbitex:ocsf" class_uid=2001 severity_id>=4
| stats count by unmapped.category unmapped.action
| sort -count

Authentication failures by user and source IP

Identifies users with repeated authentication failures and the IP addresses they originated from:

index=arbitex sourcetype="arbitex:ocsf" class_uid=3002 status="Failure"
| stats count by actor.user.uid src_endpoint.ip
| sort -count
| where count > 5

Credential detections

Lists all events where the CredInt (credential intelligence) scanner detected credentials in a request or response:

index=arbitex sourcetype="arbitex:ocsf" class_uid=2001 raw_data="credint_hit"
| table _time actor.user.uid finding_info.title actor.user.org_uid
| sort -_time

High-latency API calls

Identifies model requests with latency above 5,000 ms, aggregated by model:

index=arbitex sourcetype="arbitex:ocsf" class_uid=6003 unmapped.latency_ms>5000
| stats avg(unmapped.latency_ms) as avg_latency_ms count as request_count by unmapped.model_id
| sort -avg_latency_ms

Admin configuration changes

Shows all account change events (configuration changes, user creation, API key revocations) with full context:

index=arbitex sourcetype="arbitex:ocsf" class_uid=3004
| table _time actor.user.uid raw_data message
| sort -_time

DLP events over time (trend)

Useful for dashboards showing DLP enforcement volume over the past 24 hours:

index=arbitex sourcetype="arbitex:ocsf" class_uid=2001 earliest=-24h
| timechart span=1h count by unmapped.action

---

Troubleshooting

Events not appearing in Splunk

Symptom: No events appear in index=arbitex after platform restart and triggering test actions.

Diagnosis:

1. Check platform logs for connector errors: grep "splunk_hec" /var/log/arbitex/platform.log
2. Verify HEC endpoint is reachable from the platform host: curl -H "Authorization: Splunk <token>" <HEC_URL>
3. Check if the dead letter file is growing: wc -l /var/log/arbitex/splunk_dead_letter.jsonl

Resolution:

• If the dead letter file is growing with "Invalid token" errors, regenerate the HEC token in Splunk and update SPLUNK_HEC_TOKEN.
• If the connection is refused, verify firewall rules allow outbound traffic from the platform host to your Splunk HEC port (default 8088).
• If Splunk returns 400 with "No data" on the health check, the connector is working — this is the expected healthy response for the empty-body probe.

Events appear with wrong timestamp

Symptom: Events are indexed at the time Splunk received them rather than when they occurred on the platform.

Diagnosis: Confirm that the time field in the HEC envelope is being honored by Splunk. Check your HEC token settings: Settings > Data Inputs > HTTP Event Collector > token > Edit.

Resolution: Enable Use timestamp from event on the HEC token configuration. The connector sets the time field to the original event epoch timestamp (millisecond precision float). If Splunk is overriding this, ensure the token is not configured to use “system time.”

Events have wrong sourcetype

Symptom: Events appear in Splunk with a sourcetype other than arbitex:ocsf.

Diagnosis: Check if the HEC token has a default sourcetype configured that overrides the event payload.

Resolution: The connector hardcodes "sourcetype":"arbitex:ocsf" in the HEC payload. For this to take effect, the HEC token must be configured with Sourcetype set to blank or to arbitex:ocsf. If a token-level sourcetype overrides the payload sourcetype, update the token in Splunk.

Dead letter file growing continuously

Symptom: /var/log/arbitex/splunk_dead_letter.jsonl grows over time and does not stop.

Diagnosis: Read recent entries to understand the error: tail -n 20 /var/log/arbitex/splunk_dead_letter.jsonl | python3 -m json.tool

Resolution:

• "HTTP 403": Token is invalid or revoked. Rotate the HEC token.
• "Max retries exceeded": Splunk HEC is consistently returning 429 or 503. Check Splunk HEC queue depth and throughput limits. Consider increasing SPLUNK_HEC_FLUSH_INTERVAL to reduce delivery rate.
• "Connection refused" or "Timeout": Network path to HEC endpoint is down.

Dead letter events can be replayed manually by parsing the JSONL and re-posting the event field to the HEC endpoint:

python3 - <<'EOF'
import json, urllib.request, os

url = os.environ["SPLUNK_HEC_URL"]
token = os.environ["SPLUNK_HEC_TOKEN"]

with open("/var/log/arbitex/splunk_dead_letter.jsonl") as f:
    for line in f:
        record = json.loads(line)
        payload = json.dumps({"event": record["event"]}).encode()
        req = urllib.request.Request(url, data=payload,
            headers={"Authorization": f"Splunk {token}",
                     "Content-Type": "application/json"})
        with urllib.request.urlopen(req) as resp:
            print(resp.status, record["error"][:60])
EOF

Flush interval vs batch size trade-offs

Symptom: Events appear in Splunk in irregular bursts rather than a steady stream.

Explanation: The connector flushes when either SPLUNK_HEC_BATCH_SIZE events have accumulated or SPLUNK_HEC_FLUSH_INTERVAL seconds have elapsed, whichever comes first. During low-traffic periods, the flush interval drives delivery, so events may arrive in small batches every 5 seconds. During high-traffic periods, the batch size limit drives delivery.

Resolution: For near-real-time delivery in low-traffic environments, reduce SPLUNK_HEC_FLUSH_INTERVAL to 1–2 seconds. For high-throughput environments, increase SPLUNK_HEC_BATCH_SIZE to reduce HTTP overhead.

---

Outpost direct sink for Splunk HEC

The Outpost direct sink (outpost-0008-siem-parity) allows Arbitex Hybrid Outpost to stream audit events directly to Splunk without routing traffic through Arbitex Cloud. This is the appropriate path for air-gapped deployments or deployments where audit data must not transit Arbitex Cloud.

This section applies to Outpost deployments only. For Cloud-managed SIEM forwarding, the Platform connector configuration above applies.

Direct sink vs Platform connector

	Platform connector	Outpost direct sink
Event source	Arbitex Cloud	Arbitex Hybrid Outpost
Event format	OCSF v1.1	Raw Arbitex audit JSON
Air-gap support	No	Yes
Auth	SPLUNK_HEC_TOKEN header	SPLUNK_HEC_TOKEN header
sourcetype	arbitex:ocsf	arbitex:audit

Outpost configuration

Set the following environment variables on your Outpost deployment:

Variable	Required	Default	Description
SIEM_SINK	Yes	—	Set to splunk_hec to activate the Splunk direct sink
SPLUNK_HEC_URL	Yes	—	Full HEC collector URL, e.g. https://your-splunk:8088/services/collector/event
SPLUNK_HEC_TOKEN	Yes	—	HEC authentication token
SPLUNK_INDEX	No	arbitex	Target Splunk index name
SPLUNK_SOURCE_TYPE	No	arbitex:audit	Splunk sourcetype applied to every event
SPLUNK_VERIFY_SSL	No	true	Verify TLS certificate on the HEC endpoint
SIEM_BATCH_SIZE	No	100	Maximum number of events per HEC batch request
SIEM_FLUSH_INTERVAL_SECONDS	No	10	Maximum seconds between batch flushes

SIEM_SINK=splunk_hec
SPLUNK_HEC_URL=https://splunk.corp.example.com:8088/services/collector/event
SPLUNK_HEC_TOKEN=<YOUR_SIEM_TOKEN>
SPLUNK_INDEX=arbitex
SPLUNK_SOURCE_TYPE=arbitex:audit

Kubernetes Secret example

apiVersion: v1
kind: Secret
metadata:
  name: arbitex-outpost-siem
  namespace: arbitex
type: Opaque
stringData:
  SIEM_SINK: "splunk_hec"
  SPLUNK_HEC_URL: "https://splunk.corp.example.com:8088/services/collector/event"
  SPLUNK_HEC_TOKEN: "<YOUR_SIEM_TOKEN>"
  SPLUNK_INDEX: "arbitex"
  SPLUNK_SOURCE_TYPE: "arbitex:audit"
  SPLUNK_VERIFY_SSL: "true"
  SIEM_BATCH_SIZE: "100"
  SIEM_FLUSH_INTERVAL_SECONDS: "10"

Outpost event format

Events are delivered wrapped in the standard HEC envelope. The event field contains the raw Arbitex audit event (not OCSF-formatted):

{
  "time": 1741564800.000,
  "host": "outpost-prod-1.corp.example.com",
  "source": "arbitex:outpost",
  "sourcetype": "arbitex:audit",
  "index": "arbitex",
  "event": {
    "timestamp": "2026-03-07T12:00:00.000Z",
    "user_id": "a1b2c3d4-0001-0001-0001-000000000001",
    "action": "chat_completion",
    "model_id": "claude-sonnet-4-6",
    "provider": "anthropic",
    "token_count_input": 312,
    "token_count_output": 847,
    "cost_estimate": 0.0024,
    "latency_ms": 1840,
    "tenant_id": "org_acme",
    "hmac": "sha256:3f2a1b...",
    "previous_hmac": "sha256:7c4e9d...",
    "hmac_key_id": "key_2026_03"
  }
}

hmac, previous_hmac, and hmac_key_id are chain integrity fields. Each event’s previous_hmac must equal the hmac of the immediately preceding event for the same tenant. See Audit log verification for the full validation procedure.

Outpost dead letter path

When all retry attempts for a batch are exhausted, events are written to /var/log/arbitex/splunk_dead_letter.jsonl. To replay after the SIEM is restored:

jq -c '.event' /var/log/arbitex/splunk_dead_letter.jsonl | while read -r event; do
  curl -s -X POST "$SPLUNK_HEC_URL" \
    -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"sourcetype\": \"arbitex:audit\", \"index\": \"arbitex\", \"event\": $event}"
done