Elasticsearch SIEM connector

The Elasticsearch connector is a P0 production connector that forwards Arbitex audit events to Elasticsearch via the Bulk API. Events are formatted as OCSF v1.1 JSON documents and indexed in batches for efficiency.

How it works

Events are accumulated in an internal buffer (up to 100 events or 5 seconds, whichever comes first).
Batches are sent to POST /{index}/_bulk using NDJSON format — each event produces two lines: an index action line followed by the event document.
On HTTP 429 or 503, the connector retries with exponential backoff (1 s, 2 s, 4 s … up to 30 s max) up to max_retries attempts.
If a batch response contains "errors": true in the JSON body, the connector treats the batch as failed even if the HTTP status was 200.
If all retries fail, events are written to a dead letter JSONL file on disk.
The health check calls GET /_cluster/health and reports Healthy for green or yellow cluster status, Degraded for red.

Configuration

Set the following environment variables on your Arbitex deployment:

Variable	Required	Default	Description
`ELASTIC_URL`	Conditional	—	Elasticsearch base URL, e.g. `https://my-cluster:9200`. Required if `ELASTIC_CLOUD_ID` is not set.
`ELASTIC_CLOUD_ID`	Conditional	—	Elastic Cloud deployment ID. If set, the Elasticsearch URL is decoded automatically and `ELASTIC_URL` is not required.
`ELASTIC_API_KEY`	Conditional	—	Elasticsearch API key (`ApiKey` auth). Takes precedence over username/password.
`ELASTIC_USERNAME`	Conditional	—	Basic auth username. Used only if `ELASTIC_API_KEY` is not set.
`ELASTIC_PASSWORD`	Conditional	—	Basic auth password. Used only if `ELASTIC_API_KEY` is not set.
`ELASTIC_INDEX`	No	`arbitex-ocsf`	Target index name.
`ELASTIC_BATCH_SIZE`	No	`100`	Maximum events per batch send.
`ELASTIC_FLUSH_INTERVAL`	No	`5`	Maximum seconds between buffer flushes.
`ELASTIC_MAX_RETRIES`	No	`3`	Maximum retry attempts on transient failures.
`ELASTIC_DEAD_LETTER_PATH`	No	`/var/log/arbitex/elastic_dead_letter.jsonl`	Path for dead letter JSONL fallback.

Authentication: either ELASTIC_API_KEY or ELASTIC_USERNAME + ELASTIC_PASSWORD must be set. API key is recommended for production.

Endpoint: either ELASTIC_URL or ELASTIC_CLOUD_ID must be set.

Elastic Cloud setup

Elastic Cloud provides a hosted Elasticsearch service with built-in security. Arbitex supports both the direct URL and the Cloud ID shorthand.

Option A — Cloud ID (recommended for Elastic Cloud)

Log in to the Elastic Cloud console.
Navigate to your deployment and copy the Cloud ID from the deployment overview page.
Create an API key:
- In Kibana, go to Stack Management → Security → API Keys.
- Click Create API key.
- Set a name (e.g., arbitex-siem) and assign index privileges (write, create_index) on arbitex-ocsf*.
- Copy the key value — it is only shown once.

Set the environment variables:

ELASTIC_CLOUD_ID="my-deployment:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyQ..."
ELASTIC_API_KEY="VnVhQ2ZHY0JDZGJrUUxmUGpSbTFnOkpHdDNaLUdyUlRxM..."
ELASTIC_INDEX="arbitex-ocsf"

Option B — Direct URL

Find your Elasticsearch endpoint URL in the Elastic Cloud console under Deployments → your-deployment → Elasticsearch.
Use the same API key creation steps as above.

Set the environment variables:

ELASTIC_URL="https://abc123.us-east-1.aws.found.io:9243"
ELASTIC_API_KEY="VnVhQ2ZHY0JDZGJrUUxmUGpSbTFnOkpHdDNaLUdyUlRxM..."

Self-hosted Elasticsearch setup

Ensure your Elasticsearch cluster is reachable from the Arbitex deployment network.

Create an index with the target name (default: arbitex-ocsf):

PUT /arbitex-ocsf
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1 }
}

Create an API key with index write privileges:

POST /_security/api_key
{
  "name": "arbitex-siem",
  "role_descriptors": {
    "arbitex-writer": {
      "indices": [{
        "names": ["arbitex-ocsf*"],
        "privileges": ["write", "create_index"]
      }]
    }
  }
}

Copy the returned encoded value — this is your ELASTIC_API_KEY.

Set the environment variables:

ELASTIC_URL="https://elastic.internal:9200"
ELASTIC_API_KEY="<encoded value from above>"

If using a self-signed certificate, ensure the certificate is in the Arbitex deployment’s trust store, or configure Elasticsearch with a CA-signed certificate.

Index mapping

The connector sends raw OCSF JSON documents. Elasticsearch auto-maps fields on first ingest. For production deployments, apply an index template to enforce consistent mapping and set retention policies:

PUT /_index_template/arbitex-ocsf
{
  "index_patterns": ["arbitex-ocsf*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "class_uid":      { "type": "integer" },
        "time":           { "type": "date", "format": "epoch_millis" },
        "severity_id":    { "type": "integer" },
        "activity_id":    { "type": "integer" },
        "actor.user.uid": { "type": "keyword" },
        "actor.user.org_uid": { "type": "keyword" },
        "src_endpoint.ip": { "type": "ip" },
        "message":        { "type": "text" }
      }
    }
  }
}

Verifying the connector

In the Arbitex admin UI, navigate to Admin → SIEM. The Elastic SIEM connector row shows:

Healthy — cluster health endpoint returned green or yellow
Degraded — cluster status is red
Error — connection failed or credentials are invalid
Not configured — ELASTIC_URL/ELASTIC_CLOUD_ID or auth variables are not set

Click Send test event to send a synthetic OCSF event immediately (bypassing the buffer). In Kibana, run:

GET /arbitex-ocsf/_search
{
  "query": { "match": { "api.operation": "siem_test_event" } },
  "sort": [{ "time": "desc" }],
  "size": 1
}

Dead letter recovery

Failed batches are written to /var/log/arbitex/elastic_dead_letter.jsonl (or the path configured by ELASTIC_DEAD_LETTER_PATH). Each line is:

{
  "event": { ... },
  "error": "HTTP 503: Service Unavailable",
  "connector": "elastic",
  "timestamp": 1741564800.0
}

To replay after the cluster is restored, extract the event objects and bulk-index them:

jq -c '.event' /var/log/arbitex/elastic_dead_letter.jsonl \
  | while read event; do
      echo '{"index":{"_index":"arbitex-ocsf"}}'
      echo "$event"
    done \
  | curl -s -X POST "https://elastic.internal:9200/_bulk" \
      -H "Content-Type: application/x-ndjson" \
      -H "Authorization: ApiKey $ELASTIC_API_KEY" \
      --data-binary @-

Elasticsearch SIEM connector

How it works

Configuration

Elastic Cloud setup

Option A — Cloud ID (recommended for Elastic Cloud)

Option B — Direct URL

Self-hosted Elasticsearch setup

Index mapping

Verifying the connector

Dead letter recovery

See also