Microsoft Sentinel SIEM integration deep dive

Arbitex forwards audit events to Microsoft Sentinel via the Azure Monitor Logs Ingestion API using the SentinelConnector class (backend/app/services/siem/sentinel.py). Events are delivered as JSON arrays to a Data Collection Endpoint (DCE) and routed through a Data Collection Rule (DCR) into a custom log table in your Log Analytics workspace. All events are emitted in OCSF v1.1 format.

For a comparison of all seven Arbitex SIEM connectors, see the SIEM integration overview. For background on delivery path selection (Platform connector vs Outpost direct sink), see the SIEM integration guide.

---

Overview

The Sentinel connector operates server-side on the Arbitex Platform. It authenticates using Azure OAuth 2.0 client credentials (service principal), obtains a bearer token scoped to https://monitor.azure.com/, and posts batches of OCSF events to the Logs Ingestion API endpoint associated with your DCE.

Architecture flow:

Arbitex Platform (event source)
        |
        v
  [In-memory buffer]  <-- async background flush task
        |         \
        |          \-- flush every SENTINEL_FLUSH_INTERVAL seconds
        v
  [OAuth 2.0 token cache]
        |  (refresh when < 60 seconds remaining)
        v
  POST {dce_endpoint}/dataCollectionRules/{dcr_immutable_id}/streams/{stream_name}
       ?api-version=2023-01-01
       Authorization: Bearer <token>
       Content-Type: application/json
        |
        v
  Azure Monitor → Log Analytics workspace
        |
        v
  ArbitexOCSF_CL (custom log table)

Connector ID: sentinel

Source file: backend/app/services/siem/sentinel.py

Event format: OCSF v1.1.0, JSON array

API version: 2023-01-01

Delivery success codes: HTTP 200 or HTTP 204

---

Prerequisites

Azure prerequisites

The following Azure resources must exist before configuring the Arbitex connector.

App registration (service principal):

• An Azure AD app registration in your tenant
• A client secret generated for the registration
• The app must have the Monitoring Metrics Publisher role assigned on the DCR resource

Data Collection Endpoint (DCE):

• A DCE provisioned in the same Azure region as your Log Analytics workspace
• Network access from your Arbitex Platform host to the DCE ingestion endpoint

Data Collection Rule (DCR):

• A DCR configured to route data to your Log Analytics workspace
• A stream definition named Custom-ArbitexOCSF_CL (or your configured SENTINEL_STREAM_NAME)
• The DCR’s immutable ID is required for the ingestion URL

Log Analytics workspace:

• A custom log table named ArbitexOCSF_CL in your workspace
• The table schema must accommodate OCSF v1.1 fields

Arbitex prerequisites:

• Platform version with SentinelConnector support
• Access to set environment variables (Kubernetes secret, Helm values, or .env file)
• Write access to the dead letter directory (default /var/log/arbitex/)

Tip

The Logs Ingestion API (DCE/DCR path) is the current Microsoft-recommended path for custom log ingestion. It replaces the legacy HTTP Data Collector API. Ensure you are using a DCE-based DCR, not a legacy workspace-based data connector.

---

Step-by-step setup

1. Create an Azure AD app registration

   In the Azure portal, navigate to Azure Active Directory > App registrations > New registration.

   • Name: arbitex-sentinel-connector
   • Supported account types: Accounts in this organizational directory only
   • Redirect URI: leave blank

   After creation, note the Application (client) ID and Directory (tenant) ID — these become SENTINEL_CLIENT_ID and SENTINEL_TENANT_ID.

2. Create a client secret

   In your app registration, navigate to Certificates & secrets > Client secrets > New client secret.

   • Description: arbitex-platform
   • Expiry: 12 or 24 months (rotate before expiry)

   Copy the secret value immediately — it is only shown once. This becomes SENTINEL_CLIENT_SECRET.

3. Create a Data Collection Endpoint

   Navigate to Azure Monitor > Data Collection Endpoints > Create.

   • Name: arbitex-dce
   • Region: same as your Log Analytics workspace
   • Resource group: your preferred resource group

   After creation, note the Logs Ingestion URI (format: https://<dce-name>.<region>.ingest.monitor.azure.com). This becomes SENTINEL_DCE_ENDPOINT.

4. Create a Data Collection Rule

   Navigate to Azure Monitor > Data Collection Rules > Create.

   • Platform type: Custom
   • Name: arbitex-dcr
   • Region: same as your workspace

   Under Data sources, add a custom log source and define the stream:

   • Stream name: Custom-ArbitexOCSF_CL
   • Destination: your Log Analytics workspace

   After creation, note the Immutable ID from the DCR’s JSON view (format: dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx). This becomes SENTINEL_DCR_IMMUTABLE_ID.

5. Assign the Monitoring Metrics Publisher role

   Navigate to your DCR resource > Access control (IAM) > Add role assignment.

   • Role: Monitoring Metrics Publisher
   • Assign access to: User, group, or service principal
   • Select: your arbitex-sentinel-connector app registration

   Caution

   The role assignment must be on the DCR resource specifically, not on the subscription or resource group. Subscription-level role assignments for this role are not inherited to the Logs Ingestion API.

6. Create the custom log table

   In your Log Analytics workspace, navigate to Tables > Create > New custom log (DCR-based).

   Define the schema to accommodate OCSF v1.1 fields. The table name must be ArbitexOCSF_CL (or match your SENTINEL_STREAM_NAME with Custom- prefix stripped and _CL appended).

   Recommended minimum schema columns:

   Column name	Type	Description
   TimeGenerated	datetime	Event timestamp — required by Log Analytics
   class_uid	int	OCSF class UID
   class_name	string	Human-readable class name
   severity	string	Severity label
   actor_user_email	string	User email address
   actor_org_uid	string	Tenant/org ID
   src_ip	string	Source IP address
   message	string	Human-readable event summary

7. Set environment variables

   Kubernetes secret

   apiVersion: v1
   kind: Secret
   metadata:
     name: arbitex-siem-sentinel
     namespace: arbitex
   type: Opaque
   stringData:
     SENTINEL_TENANT_ID: "your-tenant-id"
     SENTINEL_CLIENT_ID: "your-client-id"
     SENTINEL_CLIENT_SECRET: "your-client-secret"
     SENTINEL_DCE_ENDPOINT: "https://arbitex-dce.eastus.ingest.monitor.azure.com"
     SENTINEL_DCR_IMMUTABLE_ID: "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
     SENTINEL_STREAM_NAME: "Custom-ArbitexOCSF_CL"
     SENTINEL_BATCH_SIZE: "100"
     SENTINEL_FLUSH_INTERVAL: "5"
     SENTINEL_MAX_RETRIES: "3"
     SENTINEL_DEAD_LETTER_PATH: "/var/log/arbitex/sentinel_dead_letter.jsonl"

   Helm values

   platform:
     env:
       SENTINEL_TENANT_ID: "your-tenant-id"
       SENTINEL_CLIENT_ID: "your-client-id"
       SENTINEL_CLIENT_SECRET: "your-client-secret"
       SENTINEL_DCE_ENDPOINT: "https://arbitex-dce.eastus.ingest.monitor.azure.com"
       SENTINEL_DCR_IMMUTABLE_ID: "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

   Environment file

   SENTINEL_TENANT_ID=your-tenant-id
   SENTINEL_CLIENT_ID=your-client-id
   SENTINEL_CLIENT_SECRET=ABA1FA15E000-example-client-secret
   SENTINEL_DCE_ENDPOINT=https://arbitex-dce.eastus.ingest.monitor.azure.com
   SENTINEL_DCR_IMMUTABLE_ID=dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   SENTINEL_STREAM_NAME=Custom-ArbitexOCSF_CL
   SENTINEL_BATCH_SIZE=100
   SENTINEL_FLUSH_INTERVAL=5
   SENTINEL_MAX_RETRIES=3
   SENTINEL_DEAD_LETTER_PATH=/var/log/arbitex/sentinel_dead_letter.jsonl

8. Restart the Arbitex Platform

   # Kubernetes
   kubectl rollout restart deployment/arbitex-platform -n arbitex
   
   # Docker Compose
   docker compose up -d --force-recreate platform

9. Verify event delivery

   In the Azure portal, navigate to your Log Analytics workspace > Logs. Run the following KQL query:

   ArbitexOCSF_CL
   | where TimeGenerated > ago(10m)
   | take 5

   Events typically appear within 1–3 minutes of ingestion.

---

Configuration reference

Environment variable	Required	Default	Description
SENTINEL_TENANT_ID	Yes	—	Azure AD tenant ID (Directory ID)
SENTINEL_CLIENT_ID	Yes	—	App registration Application (client) ID
SENTINEL_CLIENT_SECRET	Yes	—	App registration client secret value
SENTINEL_DCE_ENDPOINT	Yes	—	Data Collection Endpoint Logs Ingestion URI
SENTINEL_DCR_IMMUTABLE_ID	Yes	—	Data Collection Rule immutable ID (dcr-...)
SENTINEL_STREAM_NAME	No	Custom-ArbitexOCSF_CL	DCR stream name to target
SENTINEL_BATCH_SIZE	No	100	Maximum number of OCSF events per POST
SENTINEL_FLUSH_INTERVAL	No	5	Seconds between background flush cycles
SENTINEL_MAX_RETRIES	No	3	Maximum retry attempts for retriable errors
SENTINEL_DEAD_LETTER_PATH	No	/var/log/arbitex/sentinel_dead_letter.jsonl	Path to the JSONL dead letter file

---

Sample configuration

Minimal configuration (required fields only)

SENTINEL_TENANT_ID=11111111-2222-3333-4444-555555555555
SENTINEL_CLIENT_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
SENTINEL_CLIENT_SECRET=ABA1FA15E000-example-client-secret
SENTINEL_DCE_ENDPOINT=https://arbitex-dce.eastus.ingest.monitor.azure.com
SENTINEL_DCR_IMMUTABLE_ID=dcr-1234567890abcdef1234567890abcdef

Production configuration

# Azure identity
SENTINEL_TENANT_ID=11111111-2222-3333-4444-555555555555
SENTINEL_CLIENT_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
SENTINEL_CLIENT_SECRET=ABA1FA15E000-example-client-secret

# Azure Monitor ingestion
SENTINEL_DCE_ENDPOINT=https://arbitex-dce.eastus.ingest.monitor.azure.com
SENTINEL_DCR_IMMUTABLE_ID=dcr-1234567890abcdef1234567890abcdef
SENTINEL_STREAM_NAME=Custom-ArbitexOCSF_CL

# Delivery tuning
SENTINEL_BATCH_SIZE=100
SENTINEL_FLUSH_INTERVAL=5
SENTINEL_MAX_RETRIES=3

# Dead letter
SENTINEL_DEAD_LETTER_PATH=/var/log/arbitex/sentinel_dead_letter.jsonl

OAuth 2.0 token request

The connector uses the standard Azure OAuth 2.0 client credentials flow. The token URL and scope are constructed internally:

POST https://login.microsoftonline.com/{SENTINEL_TENANT_ID}/oauth2/v2.0/token

Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&client_id={SENTINEL_CLIENT_ID}
&client_secret={SENTINEL_CLIENT_SECRET}
&scope=https://monitor.azure.com/.default

The obtained token is cached and reused until it has less than 60 seconds remaining on its expiry, at which point the connector obtains a new token before the next flush cycle.

Ingestion API request

Each flush sends a JSON array of OCSF events to the following endpoint:

POST {SENTINEL_DCE_ENDPOINT}/dataCollectionRules/{SENTINEL_DCR_IMMUTABLE_ID}/streams/{SENTINEL_STREAM_NAME}?api-version=2023-01-01

Authorization: Bearer <oauth-token>
Content-Type: application/json

[
  {
    "class_uid": 2001,
    "severity_id": 4,
    "actor": {
      "user": {
        "uid": "[email protected]",
        "org_uid": "org_abc123"
      }
    },
    "finding_info": {
      "title": "Credential detected in response"
    },
    "raw_data": "credint_hit",
    "time": 1741824000123
  },
  {
    "class_uid": 6003,
    "actor": {
      "user": {
        "uid": "[email protected]"
      }
    },
    "unmapped": {
      "model_id": "gpt-4o",
      "latency_ms": 342,
      "provider": "openai"
    },
    "time": 1741824001456
  }
]

---

Authentication and token caching

The connector caches the OAuth 2.0 bearer token in memory with a 60-second safety margin. This means:

• The token is fetched once on the first flush attempt after startup.
• Subsequent flush cycles reuse the cached token if it has more than 60 seconds of validity remaining.
• When a token is within 60 seconds of expiry, the connector fetches a fresh token before the next POST.
• Token fetch failures are treated as delivery failures and trigger the retry loop.

Tip

Azure AD tokens for the https://monitor.azure.com/.default scope typically have a 1-hour lifetime. Under normal operation, the connector refreshes the token approximately once per hour.

---

Example queries

The following KQL queries target the ArbitexOCSF_CL custom log table. Field names use the Log Analytics column naming convention where nested OCSF fields are flattened with underscores and suffixed with their data type (_d for numeric, _s for string).

DLP blocks by category and action

Summarizes DLP enforcement events with severity High or Critical (severity_id 4 and 5):

ArbitexOCSF_CL
| where class_uid_d == 2001 and severity_id_d >= 4
| summarize count() by
    category_s = tostring(unmapped_category_s),
    action_s = tostring(unmapped_action_s)
| order by count_ desc

Authentication failures by user and source IP

Identifies users with repeated authentication failures and the IP addresses they originated from:

ArbitexOCSF_CL
| where class_uid_d == 3002 and status_s == "Failure"
| summarize count() by
    user_uid_s = tostring(actor_user_uid_s),
    src_ip_s = tostring(src_endpoint_ip_s)
| order by count_ desc

Credential detections

Lists all CredInt scanner events where credentials were detected in a request or response:

ArbitexOCSF_CL
| where class_uid_d == 2001 and raw_data_s == "credint_hit"
| project
    TimeGenerated,
    user_uid_s = tostring(actor_user_uid_s),
    finding_title_s = tostring(finding_info_title_s)
| order by TimeGenerated desc

High-latency API calls

Identifies model requests with latency above 5,000 ms, aggregated by model:

ArbitexOCSF_CL
| where class_uid_d == 6003 and todouble(unmapped_latency_ms_s) > 5000
| summarize
    avg_latency_ms = avg(todouble(unmapped_latency_ms_s)),
    request_count = count()
    by model_id_s = tostring(unmapped_model_id_s)
| order by avg_latency_ms desc

Admin configuration changes

Shows all account change events (configuration changes, user creation, API key revocations):

ArbitexOCSF_CL
| where class_uid_d == 3004
| project
    TimeGenerated,
    user_uid_s = tostring(actor_user_uid_s),
    raw_data_s,
    message_s
| order by TimeGenerated desc

DLP event trend over 24 hours

Useful for Sentinel workbook dashboards showing DLP enforcement volume:

ArbitexOCSF_CL
| where class_uid_d == 2001 and TimeGenerated > ago(24h)
| summarize count() by
    bin(TimeGenerated, 1h),
    action_s = tostring(unmapped_action_s)
| render timechart

---

Troubleshooting

Authentication errors (401 Unauthorized)

Symptom: Platform logs show 401 Unauthorized responses from the ingestion API. Dead letter file contains entries with "HTTP 401" errors.

Diagnosis:

1. Verify the app registration client secret has not expired in Azure AD.
2. Confirm SENTINEL_TENANT_ID and SENTINEL_CLIENT_ID match the app registration.
3. Check that the Monitoring Metrics Publisher role is assigned on the DCR (not the workspace or subscription).

Resolution:

• Rotate the client secret in Azure AD and update SENTINEL_CLIENT_SECRET.
• Re-assign the role on the DCR resource specifically.

Events not appearing in Log Analytics

Symptom: Platform logs show 200/204 responses from the ingestion API, but ArbitexOCSF_CL shows no rows.

Diagnosis: Events can take up to 3–5 minutes to appear in Log Analytics even after a successful 200/204 response. This is normal Azure Monitor ingestion latency.

Resolution:

1. Wait at least 5 minutes after the first delivery before concluding events are missing.
2. Verify the DCR stream name matches SENTINEL_STREAM_NAME exactly (case-sensitive).
3. Confirm the custom log table ArbitexOCSF_CL exists in the workspace under Tables.
4. Check Azure Monitor ingestion metrics on the DCR for successful record counts.

DCE endpoint not reachable

Symptom: Platform logs show connection errors or timeouts to the DCE endpoint. SENTINEL_DCE_ENDPOINT format may be incorrect.

Diagnosis: Verify the DCE endpoint URL by checking the DCE resource in the Azure portal > Overview > Logs Ingestion URI. The URL should end in .ingest.monitor.azure.com, not .monitor.azure.com.

Resolution: Update SENTINEL_DCE_ENDPOINT with the correct Logs Ingestion URI. Ensure outbound HTTPS (port 443) is allowed from the Arbitex Platform host to *.ingest.monitor.azure.com.

Dead letter file growing with 403 errors

Symptom: Dead letter file contains entries with "HTTP 403" errors.

Diagnosis: The service principal does not have the Monitoring Metrics Publisher role on the DCR, or the DCR immutable ID is incorrect.

Resolution:

1. In the Azure portal, navigate to your DCR > Access control (IAM). Confirm the app registration is listed as Monitoring Metrics Publisher.
2. Verify SENTINEL_DCR_IMMUTABLE_ID matches the DCR’s immutable ID (visible under DCR > Properties or JSON view, format dcr-...).

Incorrect field mapping in KQL queries

Symptom: KQL queries return no results or unexpected values even though events appear in ArbitexOCSF_CL.

Diagnosis: The Log Analytics column naming convention for nested JSON fields varies. OCSF fields like actor.user.uid are typically flattened to actor_user_uid_s but may differ based on your DCR transformation definition.

Resolution: Run a basic query to inspect the actual column names:

ArbitexOCSF_CL
| take 1
| getschema

Adjust field references in your queries to match the actual column names in your table schema.

Token refresh failures under load

Symptom: Under sustained high event volume, occasional delivery gaps appear corresponding to token refresh cycles.

Explanation: The connector refreshes the OAuth token synchronously before flushing when the cached token is about to expire. Under very high load (many events queued, frequent flushes), this can cause brief delivery pauses.

Resolution: This is expected behavior. Azure AD token requests typically complete in under 200 ms. If token refresh failures are causing dead letter accumulation, check network latency to login.microsoftonline.com from the platform host.

---

Outpost direct sink for Microsoft Sentinel

The Outpost direct sink allows Arbitex Hybrid Outpost to stream audit events directly to Sentinel without routing traffic through Arbitex Cloud. This is the appropriate path for air-gapped deployments or deployments where audit data must not transit Arbitex Cloud.

This section applies to Outpost deployments only. For Cloud-managed SIEM forwarding, use the Platform connector configuration above.

Outpost configuration

Set the following environment variables on your Outpost deployment:

Variable	Required	Default	Description
SIEM_SINK	Yes	—	Set to sentinel to activate the Sentinel direct sink
SENTINEL_WORKSPACE_ID	Yes	—	Log Analytics workspace ID
SENTINEL_DCE_URL	Yes	—	DCE Logs Ingestion URL
SENTINEL_DCR_RULE_ID	Yes	—	DCR immutable ID (begins with dcr-)
SENTINEL_STREAM_NAME	No	Custom-ArbitexAuditLogs	DCR stream name
AZURE_TENANT_ID	Yes	—	Azure AD tenant ID
AZURE_CLIENT_ID	Yes	—	App Registration client ID
AZURE_CLIENT_SECRET	Yes	—	Client secret value
SIEM_BATCH_SIZE	No	100	Maximum events per ingestion request
SIEM_FLUSH_INTERVAL_SECONDS	No	10	Maximum seconds between batch flushes

SIEM_SINK=sentinel
SENTINEL_WORKSPACE_ID=a1b2c3d4-0001-0001-0001-000000000001
SENTINEL_DCE_URL=https://arbitex-outpost-dce-xxxx.eastus.ingest.monitor.azure.com
SENTINEL_DCR_RULE_ID=dcr-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4
SENTINEL_STREAM_NAME=Custom-ArbitexAuditLogs
AZURE_TENANT_ID=a1b2c3d4-0002-0002-0002-000000000002
AZURE_CLIENT_ID=a1b2c3d4-0003-0003-0003-000000000003
AZURE_CLIENT_SECRET=<your-client-secret>

Kubernetes Secret example

apiVersion: v1
kind: Secret
metadata:
  name: arbitex-outpost-siem
  namespace: arbitex
type: Opaque
stringData:
  SIEM_SINK: "sentinel"
  SENTINEL_WORKSPACE_ID: "a1b2c3d4-0001-0001-0001-000000000001"
  SENTINEL_DCE_URL: "https://arbitex-outpost-dce-xxxx.eastus.ingest.monitor.azure.com"
  SENTINEL_DCR_RULE_ID: "dcr-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"
  SENTINEL_STREAM_NAME: "Custom-ArbitexAuditLogs"
  AZURE_TENANT_ID: "a1b2c3d4-0002-0002-0002-000000000002"
  AZURE_CLIENT_ID: "a1b2c3d4-0003-0003-0003-000000000003"
  AZURE_CLIENT_SECRET: "<your-client-secret>"
  SIEM_BATCH_SIZE: "100"
  SIEM_FLUSH_INTERVAL_SECONDS: "10"

Reference the secret in your Outpost Deployment’s envFrom:

envFrom:
  - secretRef:
      name: arbitex-outpost-siem

Outpost event format

The Outpost sink serializes each Arbitex audit event to match the ArbitexAuditLogs_CL table schema and posts events as a JSON array to the DCE endpoint:

[
  {
    "TimeGenerated": "2026-03-07T12:00:00.000Z",
    "UserId": "a1b2c3d4-0001-0001-0001-000000000001",
    "Action": "chat_completion",
    "ModelId": "claude-sonnet-4-6",
    "Provider": "anthropic",
    "PromptText": "[REDACTED]",
    "ResponseText": "[REDACTED]",
    "TokenCountInput": 312,
    "TokenCountOutput": 847,
    "TenantId": "org_acme",
    "Hmac": "sha256:3f2a1b...",
    "PreviousHmac": "sha256:7c4e9d...",
    "HmacKeyId": "key_2026_03"
  }
]

KQL queries for Outpost events

After Outpost restarts with the new configuration, use these queries in the Sentinel Logs blade:

ArbitexAuditLogs_CL
| where TimeGenerated > ago(30m)
| project TimeGenerated, UserId, Action, ModelId, TenantId
| take 20

ArbitexAuditLogs_CL
| where TimeGenerated > ago(7d)
| where Action in ("policy_block", "dlp_trigger", "dlp_redaction")
| project TimeGenerated, UserId, Action, ConversationId, TenantId
| order by TimeGenerated desc

HMAC chain integrity spot-check:

ArbitexAuditLogs_CL
| where TimeGenerated > ago(1h)
| where TenantId == "org_acme"
| project TimeGenerated, Hmac, PreviousHmac, HmacKeyId
| order by TimeGenerated asc

Verify that each row’s PreviousHmac matches the Hmac from the row immediately above it. Gaps indicate dropped events; mismatches indicate tampering.

Outpost dead letter recovery

When all retry attempts are exhausted, events are written to /var/log/arbitex/sentinel_dead_letter.jsonl. To replay after the issue is resolved:

TOKEN=$(az account get-access-token --resource https://monitor.azure.com \
  --query accessToken -o tsv)

jq -c '[.event]' /var/log/arbitex/sentinel_dead_letter.jsonl | while read -r batch; do
  curl -s -X POST "${SENTINEL_DCE_URL}/dataCollectionRules/${SENTINEL_DCR_RULE_ID}/streams/${SENTINEL_STREAM_NAME}?api-version=2023-01-01" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "$batch"
done