Attachment DLP Guide

Arbitex scans uploaded file attachments through the same multi-tier DLP pipeline used for chat messages. Files are stored with Fernet DAR (data-at-rest) encryption and may be quarantined for admin review when sensitive data is detected.

Two attachment pathways exist:

POST /api/files/upload — full file storage with DLP integration; file is persisted to disk in encrypted form.
POST /v1/attachments/scan — scan-only path; file content is discarded immediately after scanning, only structured findings are retained.

Supported file types

Both endpoints share the same MIME type whitelist defined in attachment_constants.py. Files with any other MIME type are rejected with 422 Unprocessable Entity.

Category	MIME type
Plain text	`text/plain`
CSV	`text/csv`
Markdown	`text/markdown`
Rich text	`text/rtf`
PDF	`application/pdf`
Excel (XLSX)	`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
Excel (XLS)	`application/vnd.ms-excel`
Word (DOCX)	`application/vnd.openxmlformats-officedocument.wordprocessingml.document`
PowerPoint (PPTX)	`application/vnd.openxmlformats-officedocument.presentationml.presentation`
Image (PNG)	`image/png`
Image (JPEG)	`image/jpeg`
Image (GIF)	`image/gif`
Image (WebP)	`image/webp`
HTML	`text/html`
XML	`text/xml`, `application/xml`
JSON	`application/json`
Email (EML)	`message/rfc822`
Email (MSG)	`application/vnd.ms-outlook`

File size limits

The per-org maximum file size defaults to 25 MB and is stored in the system_configs table under the key org.<org_id>.max_attachment_size_mb.

To change the limit for a specific org:

curl -s -X PUT https://api.arbitex.ai/api/v1/admin/config/org.${ORG_ID}.max_attachment_size_mb \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": "50"}'

Files exceeding the limit are rejected with 413 Request Entity Too Large before any DLP processing occurs.

Org-level feature flag

Attachment DLP scanning is enabled by default for all organizations. To disable it for a specific org:

curl -s -X PUT https://api.arbitex.ai/api/v1/admin/config/org.${ORG_ID}.attachment_dlp_enabled \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": "false"}'

When the feature is disabled, uploads to /api/files/upload continue to work but DLP scanning is skipped. The scan-only endpoint /v1/attachments/scan returns 403 Forbidden.

Upload flow: file storage with DLP (`POST /api/files/upload`)

The full upload path runs in five stages:

Upload ──► Content hash ──► DLP scan ──► Action (BLOCK / LOG / CLEAN) ──► DAR encrypt & store

Stage 1 — Content hash

A SHA-256 hash of the raw file bytes is computed before any processing. This hash is used as the content-addressed filename component: {ATTACHMENT_DIR}/{org_id}/{sha256_hex}{ext}. If a file with the same hash already exists for the org, the disk write is skipped (per-org deduplication).

Stage 2 — DLP pipeline

The file content is extracted (PDF → pages, XLSX → cells, TXT → raw) and run through the multi-tier DLP pipeline:

Tier	Detector	What it finds
1	Regex	Structured PII: SSNs, credit cards, IBANs, API keys, etc.
2	NER (Presidio/GLiNER)	Unstructured PII: names, addresses, emails, phone numbers
3	DeBERTa validator	High-confidence re-ranking of tier-1 and tier-2 findings

Stage 3 — DLP action

The pipeline result determines what happens next:

Outcome	Condition	Result
BLOCK	Any finding with action `BLOCK`	`422 Unprocessable Entity` — file not stored; audit event written
QUARANTINE (LOG)	Findings present, none blocking	File stored with `quarantine_until` timestamp set (default +30 days)
CLEAN	No findings	File stored without quarantine flag

The quarantine duration is configurable per org via org.<org_id>.attachment_quarantine_days (default: 30).

Stage 4 — DAR encryption

Files that pass the DLP check (CLEAN or LOG) are encrypted with Fernet symmetric encryption before being written to disk. The encryption key is loaded from the ATTACHMENT_ENCRYPTION_KEY environment variable (or the configured secrets backend). Raw file bytes are never written to disk.

Stage 5 — FileUpload record

A FileUpload database row is created with:

Field	Value
`id`	UUID for the upload
`org_id`	Org UUID from the authenticated user
`user_id`	Uploader UUID
`content_hash`	SHA-256 of raw bytes
`quarantine_until`	Deadline for auto-purge (LOG outcome only)
`reviewed_at`	`null` until admin reviews

Scan-only flow (`POST /v1/attachments/scan`)

Use the scan endpoint to check whether a file contains sensitive data without storing it. File bytes are read, extracted, scanned, and then discarded — only the structured AttachmentScanResult row is persisted to the database.

curl -s -X POST https://api.arbitex.ai/v1/attachments/scan \
  -H "Authorization: Bearer $ARBITEX_API_KEY" \
  -F "file=@/path/to/report.pdf"

Response 200 OK:

{
  "scan_id": "b3c4d5e6-0000-0000-0000-000000000001",
  "org_id": "a1b2c3d4-0000-0000-0000-000000000001",
  "filename": "report.pdf",
  "mime_type": "application/pdf",
  "file_size_bytes": 184320,
  "page_count": 3,
  "findings": [
    {
      "entity_type": "CREDIT_CARD",
      "text": "4111 1111 1111 1111",
      "confidence": 0.99,
      "page_number": 2,
      "cell_reference": null,
      "dlp_tier": 1
    },
    {
      "entity_type": "EMAIL_ADDRESS",
      "text": "[email protected]",
      "confidence": 0.95,
      "page_number": 1,
      "cell_reference": null,
      "dlp_tier": 2
    }
  ],
  "finding_count": 2,
  "compliance_flags": ["PCI-DSS", "GDPR"],
  "scanned_at": "2026-03-16T14:22:00Z",
  "extraction_method": "pdfminer",
  "dlp_tiers_run": [1, 2]
}

AttachmentScanResult fields

Field	Type	Description
`scan_id`	UUID string	Unique scan result identifier
`org_id`	UUID string	Organization that submitted the scan
`filename`	string	Original filename as provided by the uploader
`mime_type`	string	MIME type of the uploaded file
`file_size_bytes`	int	Raw file size in bytes
`page_count`	int	Number of pages or logical sections extracted
`findings`	`AttachmentFinding[]`	All DLP findings across all pages
`finding_count`	int	Total number of findings
`compliance_flags`	`string[]`	Compliance frameworks triggered (HIPAA, PCI-DSS, GDPR)
`scanned_at`	ISO-8601	UTC timestamp when the scan completed
`extraction_method`	string	Text extraction method used (e.g. `pdfminer`, `openpyxl`)
`dlp_tiers_run`	`int[]`	DLP tiers that executed (e.g. `[1, 2]`)

AttachmentFinding fields

Field	Type	Description
`entity_type`	string	Canonical entity class in uppercase (e.g. `CREDIT_CARD`, `PERSON`)
`text`	string	Detected text fragment (may be redacted per org policy)
`confidence`	float	Detection confidence in `[0.0, 1.0]`
`page_number`	int	1-based page index; always `1` for flat files
`cell_reference`	string \| null	Spreadsheet cell reference (e.g. `B3`) when applicable
`dlp_tier`	int	Detection tier: `1` = regex, `2` = NER, `3` = DeBERTa

Compliance flag mapping

Compliance flags are derived from the entity types found in the scan:

Framework	Triggered by entity types
`HIPAA`	`us_ssn`, `medical_record`, `health_info`, `PERSON`, `EMAIL_ADDRESS`, `date_of_birth`, `IP_ADDRESS`, `phone_number`
`PCI-DSS`	`credit_card`, `CREDIT_CARD`, `iban`, `IBAN_CODE`, `bank_account`
`GDPR`	`PERSON`, `EMAIL_ADDRESS`, `PHONE_NUMBER`, `IP_ADDRESS`, `date_of_birth`, `national_id`

Excel scanning: cell-level findings

For XLSX and XLS files, the extractor operates at cell granularity. When a finding occurs in a spreadsheet cell, the cell_reference field contains the standard A1-style reference:

{
  "entity_type": "US_SSN",
  "text": "123-45-6789",
  "confidence": 0.99,
  "page_number": 1,
  "cell_reference": "D14",
  "dlp_tier": 1
}

This allows reviewers to pinpoint the exact cell containing sensitive data.

Quarantine administration

Files quarantined because of DLP findings are held pending admin review. Admins can list and review quarantined files using the admin endpoints.

List quarantined files

curl -s "https://api.arbitex.ai/api/v1/admin/attachments/quarantine" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Query parameters:

Parameter	Type	Default	Description
`org_id`	UUID	—	Filter by organization
`user_id`	UUID	—	Filter by uploading user
`limit`	int	50	Results per page (1–200)
`offset`	int	0	Pagination offset

Response 200 OK:

{
  "items": [
    {
      "id": "f1e2d3c4-0000-0000-0000-000000000001",
      "original_name": "payroll-q1.xlsx",
      "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "size_bytes": 52480,
      "user_id": "aabbccdd-0000-0000-0000-000000000001",
      "org_id": "a1b2c3d4-0000-0000-0000-000000000001",
      "quarantine_until": "2026-04-15T10:22:00Z",
      "reviewed_by": null,
      "reviewed_at": null,
      "created_at": "2026-03-16T10:22:00Z"
    }
  ],
  "total": 1,
  "limit": 50,
  "offset": 0
}

Items are ordered by quarantine_until ascending — files closest to their auto-purge deadline appear first.

Review a quarantined file

# Retain the file (mark as reviewed, keep in storage)
curl -s -X POST \
  "https://api.arbitex.ai/api/v1/admin/attachments/${FILE_ID}/review" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"retain": true}'

# Purge the file (mark for deletion on next purge cycle)
curl -s -X POST \
  "https://api.arbitex.ai/api/v1/admin/attachments/${FILE_ID}/review" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"retain": false}'

Request body:

Field	Type	Default	Description
`retain`	boolean	`true`	`true` = keep file; `false` = schedule for deletion

Response 200 OK:

{
  "file_id": "f1e2d3c4-0000-0000-0000-000000000001",
  "retain": true,
  "reviewed_at": "2026-03-16T15:30:00Z"
}

When retain: false, the file’s quarantine_until is set to now and reviewed_at is cleared so the background purge task removes it on the next cycle. The reviewed_at field in the response will be null.

Every review decision writes an audit event with action attachment.quarantine_reviewed.

Auto-purge

Quarantined files that are not reviewed before their quarantine_until deadline are automatically removed by the background scheduler. The purge task deletes files where quarantine_until < now AND reviewed_at IS NULL.

Error reference

Status	Endpoint	Condition
`403 Forbidden`	`/v1/attachments/scan`	`attachment_dlp_enabled = false` for the org
`403 Forbidden`	`/api/v1/admin/attachments/*`	Caller is not an admin
`404 Not Found`	`/api/v1/admin/attachments/{id}/review`	No quarantined file with given ID
`413 Request Entity Too Large`	Both	File exceeds org-level size limit
`422 Unprocessable Entity`	Both	Unsupported MIME type
`422 Unprocessable Entity`	`/api/files/upload`	DLP BLOCK action triggered
`500 Internal Server Error`	Both	Text extraction or DLP pipeline failure

Text normalization

Before DLP scanning, file text content undergoes normalization to defeat obfuscation techniques:

Bidirectional text (bidi): Unicode right-to-left overrides and directional control characters are stripped to prevent visual obfuscation of sensitive data (e.g., reversed credit card numbers).
NFKC normalization: Unicode text is normalized to NFKC form, collapsing visually similar characters (e.g., full-width digits ０１２３ → 0123) so DLP patterns match regardless of encoding tricks.

Download and decrypt

Stored attachments can be downloaded via the authenticated download endpoint:

GET /api/files/{file_id}/download
Authorization: Bearer <token>

The file is decrypted from its DAR-encrypted form on the fly and streamed to the client. The response includes an X-Quarantine-Status header:

Value	Meaning
`clean`	No DLP findings
`quarantined`	DLP findings present, pending admin review
`reviewed`	Admin has reviewed and retained the file

Orphaned files (uploaded but not linked to any message or quarantine record) are periodically cleaned by the background orphan cleanup task.

Message linking

Files can be attached to conversation messages using the file_ids field when sending a message:

POST /api/conversations/{conversation_id}/messages
Authorization: Bearer <token>
Content-Type: application/json

{
  "content": "Please review this document",
  "file_ids": ["file-uuid-1", "file-uuid-2"]
}

Linked files are accessible via the message’s attachments array in API responses.

File metadata

GET /api/files/{file_id}/metadata
Authorization: Bearer <token>

Returns file metadata (name, size, MIME type, DLP scan status) without downloading the file content.

Release and delete

Admins can release quarantined files or permanently delete them:

POST /api/v1/admin/attachments/{file_id}/release
POST /api/v1/admin/attachments/{file_id}/delete

Release removes the quarantine flag and makes the file accessible to users. Delete permanently removes the file from storage.

AI context injection

When files are attached to a conversation, their text content is automatically injected into the AI model context. The MAX_ATTACHMENT_TOKENS system config key (default: 4096) controls the maximum number of tokens extracted from attached files per message.

Content exceeding the token limit is truncated with a notice appended to the context. This allows the AI to reference file contents while preventing context window overflow from large documents.

Conversation attachments (Epic T Phase 3)

Conversation-level file attachments extend the per-message attachment model:

Chunked upload: large files can be uploaded in chunks via a multi-part upload protocol for reliability
Thumbnails: image attachments generate thumbnails for preview in the chat UI
SSE scan status: the DLP scan progress is streamed to the client via Server-Sent Events, providing real-time feedback on scan completion
File search: GET /api/files?search=<query> searches across file names and extracted text content
Cloud Portal UI: the Cloud Portal chat interface supports drag-and-drop file attachment with inline DLP status indicators

Attachment DLP Guide

Supported file types

File size limits

Org-level feature flag

Upload flow: file storage with DLP (POST /api/files/upload)

Stage 1 — Content hash

Stage 2 — DLP pipeline

Stage 3 — DLP action

Stage 4 — DAR encryption

Stage 5 — FileUpload record

Scan-only flow (POST /v1/attachments/scan)

AttachmentScanResult fields

AttachmentFinding fields

Compliance flag mapping

Excel scanning: cell-level findings

Quarantine administration

List quarantined files

Review a quarantined file

Auto-purge

Error reference

Text normalization

Download and decrypt

Message linking

File metadata

Release and delete

AI context injection

Conversation attachments (Epic T Phase 3)

See also

Upload flow: file storage with DLP (`POST /api/files/upload`)

Scan-only flow (`POST /v1/attachments/scan`)