Data Migration Guide

This guide covers all supported migration paths between Arbitex deployment modes. Use it when moving between self-hosted and cloud-managed deployments, adding or reconfiguring outposts, or transitioning between air-gap and connected operation.

For a dedicated walkthrough of the self-hosted to cloud path, see Self-hosted to cloud migration.

---

1. Migration paths overview

The following migration paths are fully supported. Each path has a dedicated section below.

From	To	Supported	Notes
Self-hosted	Cloud-managed	Yes	Full config + data migration
Cloud-managed	Self-hosted	Yes	Config export; user data requires DB migration
Self-hosted (no outpost)	Self-hosted + outpost	Yes	Add outpost; platform unchanged
Cloud-managed	Cloud + outpost	Yes	Register outpost via cloud portal
Air-gap outpost	Connected outpost	Yes	Remove OUTPOST_AIRGAP flag, configure connectivity
Connected outpost	Air-gap outpost	Yes	Stage policies/models locally, set OUTPOST_AIRGAP=true

---

2. Data portability inventory

The table below lists every data type that can be exported, imported, or archived. Review it before starting any migration to ensure you account for all necessary data.

Data Type	Export Method	Import Method	Scope
Org configuration (20 domains)	GET /api/v1/admin/orgs/{org_id}/config/export	POST /api/v1/admin/orgs/{org_id}/config/import	DLP rules, compliance bundles, policy templates, routing rules, enterprise entitlements, org metadata, group DLP configs, group model access, group compliance bundles, content filters, quotas, model catalog (export-only), kill switch state, fallback chains, webhooks (excl. HMAC secrets), alert rules, SAML IdP configs (export-only), IP allowlist, retention policies
Compliance bundles	GET /api/v1/admin/compliance-bundles-export/	POST /api/v1/admin/compliance-bundles-export/import	Bundle name, regulatory_framework, seed_rule_mappings. Validates entity types against canonical taxonomy. Mode: skip or replace. Export version 1.0.
Conversations (all)	POST /api/conversations/export-all	Manual import (no API)	ZIP of all user conversations in JSON or Markdown format. JSON mode includes token stats.
Single conversation	GET /api/conversations/{id}/export	—	JSON or Markdown
Audit logs	POST /api/v1/admin/audit/export	— (archive only)	HMAC-SHA256 signed JSON package, up to 90-day window, streaming for >10k records. Chain status: intact / broken / disabled.
Usage records	GET /api/v1/admin/orgs/{org_id}/usage/export?format=csv	— (archive only)	CSV: date, request_count, token_count, cost_usd, dlp_events, provider_breakdown. Max 366-day window.
Outpost audit buffer	outpost audit export CLI	— (archive only)	JSONL from local audit_buffer/, optional HMAC chain verification.
Outpost alert rules	outpost alerts export CLI	— (archive only)	Alertmanager/Prometheus YAML

Note

Config export schema is versioned. The current version is 2.0. Importing a v1.0 export into a v2.0 instance is supported — new domains will be empty. Importing a v2.0 export into a v1.0 instance is not supported.

---

3. Self-hosted to cloud migration

For the complete procedure, follow the dedicated guide: Self-hosted to cloud migration.

Key steps at a glance:

1. Export org config from the self-hosted platform — GET /api/v1/admin/orgs/{org_id}/config/export.
2. Provision org in the cloud portal — create the org and record org_id.
3. Import config into the cloud org — POST /v1/orgs/{org_id}/config/import. The cloud proxy saves a pre-import snapshot to organizations.config_snapshot and accepts schema versions 1.0, 1.0.0, 2.0, and 2.0.0.
4. Re-create users manually — config export does not include user accounts or billing data.
5. Update DNS / proxy config to point to the cloud endpoint.
6. Archive the self-hosted database for regulatory retention before decommissioning.

---

4. Cloud to self-hosted migration

1. Deploy the self-hosted platform.

   Follow the Deployment guide to stand up a Docker Compose or Kubernetes instance. Do not start the application until migrations complete.

2. Run database migrations.

   python scripts/run-migrations.py

3. Complete first-time setup.

   curl -s -X POST https://your-platform-host/api/setup \
     -H "Content-Type: application/json" \
     -d '{
       "admin_email": "[email protected]",
       "admin_password": "change-me",
       "org_name": "Acme Corp"
     }'

   The /api/setup endpoint is guarded by a zero-user check — it returns 409 if any users already exist.

4. Export config from cloud.

   curl -s -H "Authorization: Bearer $CLOUD_TOKEN" \
     "https://your-platform-host/v1/orgs/$ORG_ID/config/export" \
     -o org-config-export.json

5. Import config into self-hosted.

   curl -s -X POST https://your-platform-host/api/v1/admin/orgs/$ORG_ID/config/import \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H "Content-Type: application/json" \
     -d @org-config-export.json

6. Re-create users manually.

   User accounts are not included in config exports. Provision users via the admin UI or the Users API (POST /api/v1/admin/users).

7. Archive cloud data for compliance.

   Before decommissioning the cloud org, export and archive:

   • Conversations: POST /api/conversations/export-all
   • Audit logs: POST /api/v1/admin/audit/export (90-day window per request; run multiple windows to cover the full retention period)
   • Usage records: GET /api/v1/admin/orgs/{org_id}/usage/export?format=csv

8. Update DNS / proxy config to route traffic to the self-hosted instance.

9. Decommission the cloud org once traffic is confirmed routed to self-hosted and archived data is verified.

Caution

Config import does not include: user accounts, conversations, usage records, billing data, or HMAC secrets. Migrate or recreate each of these separately.

---

5. Adding an outpost

Cloud-managed + outpost

1. Register the outpost in the cloud portal.

   Navigate to Settings → Outposts → Register outpost. The portal issues an OUTPOST_ID and a certificate bundle (cert, key, CA).

2. Prepare the outpost host.

   Prerequisites:

   • Docker Engine 24+ with Compose V2
   • curl installed
   • 5 GB free disk
   • Port 8300 free and accessible from the platform

3. Run the outpost installer.

   bash install.sh

   The interactive installer prompts for outpost ID, platform URL, and cert paths. It writes .env and docker-compose.outpost.yml.

4. Review .env before starting.

   # Required env vars
   OUTPOST_ID=op_xxxxxxxxxxxxxxxx
   PLATFORM_MANAGEMENT_URL=https://your-platform-host
   AUDIT_HMAC_KEY=<32-byte hex key>
   
   # Certificate paths (mounted into the container)
   OUTPOST_CERT_PATH=/etc/arbitex/outpost/tls.crt
   OUTPOST_KEY_PATH=/etc/arbitex/outpost/tls.key
   OUTPOST_CA_PATH=/etc/arbitex/outpost/ca.crt

5. Start the outpost.

   docker compose -f docker-compose.outpost.yml up -d

6. Verify operation.

   # Health check
   curl http://localhost:8300/health
   
   # Expected
   {"status":"ok","outpost_id":"op_xxxxxxxxxxxxxxxx"}

   Confirm the heartbeat appears in the platform logs within 60 seconds.

Self-hosted + outpost

1. Register the outpost in the self-hosted admin UI.

   Navigate to Admin → Outposts → Register outpost. Record the issued OUTPOST_ID.

2. Provision TLS certificates manually.

   Cloud orgs issue certs automatically. For self-hosted, provision certs from your internal CA or a public CA, and place them at the paths you will specify in .env.

3. Run the outpost installer on the outpost host.

   bash install.sh

4. Configure .env.

   OUTPOST_ID=op_xxxxxxxxxxxxxxxx
   PLATFORM_MANAGEMENT_URL=https://your-platform-host
   AUDIT_HMAC_KEY=<32-byte hex key>
   
   OUTPOST_CERT_PATH=/etc/arbitex/outpost/tls.crt
   OUTPOST_KEY_PATH=/etc/arbitex/outpost/tls.key
   OUTPOST_CA_PATH=/etc/arbitex/outpost/ca.crt

5. Start the outpost.

   docker compose -f docker-compose.outpost.yml up -d

6. Verify operation.

   curl http://localhost:8300/health

   Confirm the heartbeat appears in the self-hosted platform logs.

---

6. Air-gap transitions

Connected → air-gap

1. Stage the policy bundle locally.

   Download the current policy bundle from the platform and place it at AIRGAP_POLICY_PATH (default: /opt/arbitex/policies/policy_bundle.json).

   curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
     "https://your-platform-host/api/v1/admin/policy-bundle" \
     -o /opt/arbitex/policies/policy_bundle.json

2. Stage the DLP model (if DLP is enabled).

   If your outpost uses the DeBERTa DLP model, download the ONNX model files and place them at AIRGAP_MODEL_PATH (default: /opt/arbitex/models/).

   # Copy model files from a connected host or USB media
   cp -r /media/usb/arbitex-models/* /opt/arbitex/models/

3. Set OUTPOST_AIRGAP=true in .env.

   .env

   OUTPOST_AIRGAP=true
   AIRGAP_POLICY_PATH=/opt/arbitex/policies/policy_bundle.json
   AIRGAP_MODEL_PATH=/opt/arbitex/models/

4. Restart the outpost.

   docker compose -f docker-compose.outpost.yml down
   docker compose -f docker-compose.outpost.yml up -d

5. Verify air-gap operation.

   Confirm the outpost loads policy from the local path and makes no outbound connections. Check logs:

   docker compose -f docker-compose.outpost.yml logs outpost | grep -E "airgap|policy"

   Expected: policy loaded from local path, no connecting to platform entries.

Air-gap → connected

1. Configure network connectivity from the outpost host to the platform endpoint. Verify reachability:

   curl -s https://your-platform-host/health

2. Remove OUTPOST_AIRGAP from .env (or set it to false).

   OUTPOST_AIRGAP=true
   OUTPOST_AIRGAP=false

3. Set platform connectivity vars in .env.

   PLATFORM_MANAGEMENT_URL=https://your-platform-host
   OUTPOST_CERT_PATH=/etc/arbitex/outpost/tls.crt
   OUTPOST_KEY_PATH=/etc/arbitex/outpost/tls.key
   OUTPOST_CA_PATH=/etc/arbitex/outpost/ca.crt

4. Restart the outpost.

   docker compose -f docker-compose.outpost.yml down
   docker compose -f docker-compose.outpost.yml up -d

5. Verify connectivity.

   Confirm the outpost heartbeat registers with the platform and policy sync resumes:

   docker compose -f docker-compose.outpost.yml logs outpost | grep -E "heartbeat|policy sync"

   Expected: heartbeat sent, policy sync complete.

---

7. Config import behavior

Understanding import mechanics prevents surprises during migration.

Platform import — POST /api/v1/admin/orgs/{org_id}/config/import:

• Admin-only endpoint — requires an admin-scoped token.
• Backward compatible across schema versions 1.0 and 2.0. Domains present in v2.0 but absent in a v1.0 export are left empty (not deleted).
• Transactional — if any domain fails validation, the entire import is rolled back.

Cloud import — POST /v1/orgs/{org_id}/config/import:

• Saves a pre-import snapshot to the organizations.config_snapshot column before applying changes.
• Accepts schema versions 1.0, 1.0.0, 2.0, and 2.0.0.
• Applies DLP rule changes transactionally.

Cloud rollback — POST /v1/orgs/{org_id}/config/rollback:

• Restores DLP rules from the snapshot saved during the most recent import.
• Returns 400 if no prior import has been performed (no snapshot exists).

Compliance bundle import — POST /api/v1/admin/compliance-bundles-export/import:

• mode=skip (default) — skips any bundle whose name already exists. Safe for incremental migrations.
• mode=replace — overwrites existing bundles by name. Use with care in production.
• Validates all entity types against the canonical taxonomy. Bundles referencing unknown entity types are rejected.

# Import compliance bundles, replacing existing
curl -s -X POST https://your-platform-host/api/v1/admin/compliance-bundles-export/import \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode": "replace", "bundles": [...]}'

Caution

Cloud rollback covers DLP rules only, not the full 20-domain config. For a full config rollback, restore from a database backup taken before the import.

---

8. Database migration (PostgreSQL)

Full database migration is required when moving all data — including users, conversations, and usage records — between instances. Config import alone is insufficient for this scenario.

Export from source instance:

# Platform database
pg_dump -Fc -d arbitex_platform -f platform-migration-$(date +%Y%m%d).dump

# Cloud database (if self-hosting the cloud component)
pg_dump -Fc -d arbitex_cloud -f cloud-migration-$(date +%Y%m%d).dump

Verify schema revision before import:

Ensure the target instance has run all migrations to the same revision as the source.

python scripts/run-migrations.py --check

If the target is behind, run migrations first:

python scripts/run-migrations.py

Import to target instance:

# Platform database
pg_restore -d arbitex_platform --clean --if-exists platform-migration-YYYYMMDD.dump

# Cloud database
pg_restore -d arbitex_cloud --clean --if-exists cloud-migration-YYYYMMDD.dump

Outpost has no database. The outpost is stateless. Persistent data lives in two flat-file locations:

• audit_buffer/ — local JSONL audit event buffer. Copy as-is to transfer buffered events.
• policy_cache/ — cached policy bundle. Regenerated on next sync; copy is optional.

# Archive outpost local state before decommissioning
tar -czf outpost-local-state-$(date +%Y%m%d).tar.gz audit_buffer/ policy_cache/

---

9. Post-migration verification

Run all checks in the table below before declaring the migration complete.

Check	Command	Expected
API health	curl /health	{"status": "ok"}
Config domain count	GET /api/v1/admin/orgs/{org_id}/config/export	20 domains present
DLP rules loaded	GET /api/v1/admin/dlp-rules	Rule count matches source
Compliance bundles	GET /api/v1/admin/compliance-bundles	Bundle count matches source
Audit chain	POST /api/v1/admin/audit/export	chain_status: "intact"
Outpost heartbeat	Platform logs	Version reported, policy synced
User login	Test login flow	Admin and regular user both succeed

Spot-check DLP rule parity:

# Count rules on source (before migration)
SOURCE_COUNT=$(curl -s -H "Authorization: Bearer $SOURCE_TOKEN" \
  https://source-host/api/v1/admin/dlp-rules | jq '.total')

# Count rules on target (after migration)
TARGET_COUNT=$(curl -s -H "Authorization: Bearer $TARGET_TOKEN" \
  https://target-host/api/v1/admin/dlp-rules | jq '.total')

echo "Source: $SOURCE_COUNT  Target: $TARGET_COUNT"

Verify audit chain integrity:

curl -s -X POST https://your-platform-host/api/v1/admin/audit/export \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "start_date": "2025-01-01",
    "end_date": "2025-03-31"
  }' | jq '.chain_status'

Expected output: "intact". A "broken" chain status indicates log tampering or a gap in event records — investigate before signing off on the migration.

---

Related pages

• Upgrade guide
• Self-hosted to cloud migration
• Deployment guide
• Outpost deployment
• Air-gap deployment
• Audit log management
• Config backup and restore