CredInt microservice deployment guide

The Credential Intelligence (CredInt) microservice checks candidate credentials extracted from prompts against a corpus of known-compromised secrets sourced from HaveIBeenPwned (HIBP) and partner breach feeds. It ships in two deployment modes: a standalone FastAPI microservice embedded in the Arbitex platform stack, and an in-process bloom filter scanner embedded in the Outpost proxy (Tier 4 DLP).

For policy configuration and user-facing setup, see the Outpost Credential Intelligence guide.

---

Architecture overview

Deployment modes

CredInt operates in one of two modes depending on where credential checking occurs:

Platform microservice mode — A standalone FastAPI service (port 8202) that the platform API calls over HTTP. It maintains a two-tier lookup structure: a hot cache dictionary (top entries by frequency, loaded into a Python dict for O(1) lookup) backed by a FastBloomFilter (numpy + mmh3) for entries outside the hot cache. The platform client posts a SHA-1 prefix to /v1/check and receives a frequency bucket in response. If the corpus or filter is unavailable, the service returns disabled and the platform fails open.

Outpost embedded mode — An ARBF binary bloom filter loaded in-process inside the Outpost proxy. Checks run against the in-memory filter with zero network calls at request time. The filter is refreshed from a CDN endpoint on a configurable schedule (default 24 hours) using conditional HTTP GET with ETag validation and atomic file swap. If the CDN is unreachable or the filter file is absent, the Outpost fails open and continues processing without CredInt.

Privacy guarantee

Credential cleartext is never logged, stored, or returned by any component. The extraction pipeline SHA-1 hashes each candidate immediately and discards the cleartext. Only the first 8 hexadecimal characters of the SHA-1 hash (sha1_prefix) appear in audit entries, added by migration 041 across 6 audit columns.

Fail-open design

Both modes are designed to fail open:

• Platform mode: if CREDINT_ENABLED=false or the corpus files are missing at startup, /health returns status: disabled and all /v1/check calls return not_found.
• Outpost mode: if CREDINT_BLOOM_PATH is absent or the CDN is unreachable at startup, CredInt is skipped in the DLP pipeline. The Outpost continues to process requests through all other tiers.

---

Platform microservice deployment

Docker deployment

Run the CredInt microservice as a sidecar alongside the platform API container:

docker run -d \
  --name credint \
  --user 1001:1001 \
  -p 8202:8202 \
  -v /data/credint/corpus:/corpus:ro \
  -e CREDINT_CORPUS_PATH=/corpus \
  -e CREDINT_ENABLED=true \
  -e CREDINT_PORT=8202 \
  arbitex/credint:latest

The service runs as non-root UID 1001. Mount the corpus directory read-only. The corpus volume must contain hot_cache.pkl and corpus.bloom (see HIBP corpus management).

Environment variables

Variable	Default	Description
CREDINT_CORPUS_PATH	/corpus	Directory containing hot_cache.pkl, corpus.bloom, and corpus-meta.json.
CREDINT_ENABLED	false	Set to true to activate corpus loading and checking. When false, the service starts but returns disabled on all checks.
CREDINT_PORT	8202	TCP port the FastAPI service listens on.

Platform client configuration

The platform API connects to the CredInt service using these environment variables:

Variable	Default	Description
CREDINT_SERVICE_URL	http://credint:8202	Base URL of the CredInt microservice.
CREDINT_SERVICE_TIMEOUT	3.0	HTTP request timeout in seconds. Requests exceeding this threshold count as failures for circuit breaker purposes.

Circuit breaker

The platform client implements a circuit breaker to protect against CredInt service unavailability:

• Threshold: 3 consecutive failures (timeouts or HTTP errors) open the circuit.
• Reset window: 60 seconds in the open state before transitioning to half-open.
• Half-open probe: one test request is attempted; success closes the circuit, failure restarts the 60-second window.
• Fail-open: while the circuit is open, all credential checks return not_found immediately without calling the service.

Corpus files

The corpus directory must contain three files produced by the build script (see HIBP corpus management):

File	Description
hot_cache.pkl	Python dict mapping SHA-1 → frequency count for the top 100 million entries. Serialized with pickle. Loaded entirely into memory (~6–10 GB).
corpus.bloom	FastBloomFilter instance serialized with pickle (numpy bit array + mmh3 hash seeds 0–9). Covers 857 million items at 0.1% FPR (~1.54 GB on disk).
corpus-meta.json	Build metadata: snapshot date, item count, FPR target, build duration. Surfaced by /health.

API endpoints

POST /v1/check

Submit a SHA-1 prefix for lookup. The service checks the hot cache first, then the bloom filter.

curl -s -X POST http://credint:8202/v1/check \
  -H "Content-Type: application/json" \
  -d '{"sha1_prefix": "5baa61e4"}'

Response:

{
  "found": true,
  "frequency_bucket": "critical",
  "credint_enabled": true
}

GET /health

Returns corpus load status and service health.

curl -s http://credint:8202/health

Response:

{
  "status": "ok",
  "credint_enabled": true,
  "corpus_loaded": true,
  "hot_cache_size": 100000000,
  "filter_type": "FastBloomFilter",
  "snapshot_date": "2025-11-01"
}

status values:

Value	Meaning
ok	Corpus loaded, service fully operational.
degraded	Corpus partially loaded (e.g., hot cache missing, only bloom filter available).
disabled	CREDINT_ENABLED=false or corpus files not found at startup.

Frequency buckets (platform mode)

The platform maps raw frequency counts to buckets used in policy evaluation and audit logs:

Bucket	Frequency threshold
critical	> 1,000,000 occurrences
high	> 100,000 occurrences
medium	> 10,000 occurrences
low	≤ 10,000 occurrences

---

Outpost embedded deployment

Environment variables

Configure CredInt behavior on the Outpost container using these variables:

Variable	Default	Description
CREDINT_ENABLED	true	Enable or disable the CredInt DLP tier. Set to false to skip entirely without removing the filter file.
CREDINT_BLOOM_PATH	—	Filesystem path to the ARBF bloom filter file (e.g., /app/credint/credint.bf). Required when not using CDN refresh.
CREDINT_CDN_URL	—	URL of the CDN endpoint serving the ARBF filter. When set, the Outpost fetches and refreshes the filter on the configured interval. Set to empty string for air-gap mode.
CREDINT_REFRESH_INTERVAL_SECONDS	86400	CDN refresh interval in seconds (default 24 hours).
CREDINT_KANON_ENABLED	false	Enable optional k-anonymity secondary check via HIBP API to confirm bloom filter hits. Disabled by default — only enable when HIBP API access is available and acceptable latency is tolerable.
CREDINT_KANON_URL	—	Base URL for the HIBP k-anonymity API endpoint. Required when CREDINT_KANON_ENABLED=true.

ARBF binary format

The Arbitex Bloom Filter (ARBF) format is the binary container used for Outpost filter distribution. Each file begins with a 62-byte header:

Field	Size	Description
Magic	4 bytes	ASCII ARBF
Version	2 bytes	Format version (uint16, little-endian)
Snapshot date	8 bytes	ISO 8601 date string (YYYYMMDD)
Item count	8 bytes	Number of items indexed (uint64, little-endian)
FPR target	8 bytes	Target false positive rate as IEEE 754 float64
k	4 bytes	Number of hash functions (uint32, little-endian)
m	8 bytes	Bit array size in bits (uint64, little-endian)
Reserved	20 bytes	Zero-padded, reserved for future use

The header is followed immediately by the packed bit array (ceil(m / 8) bytes). The Outpost uses double hashing derived from SHA-256 of the candidate credential to compute the k bit positions.

CDN refresh process

When CREDINT_CDN_URL is set, the Outpost refresh worker runs on the configured interval:

1. Sends a conditional GET to the CDN URL with If-None-Match: <current_etag>.
2. On HTTP 304 (Not Modified), skips download and logs credint_refresh: not_modified.
3. On HTTP 200, streams the response body to a temporary file.
4. Validates the ARBF magic bytes and header integrity.
5. Checks version: if the new filter version is lower than the current loaded version, rejects the download (no downgrade).
6. Atomically swaps the temporary file into CREDINT_BLOOM_PATH using os.replace().
7. Reloads the filter into memory and updates loaded_at and snapshot_date.
8. Logs credint_refresh: updated, entry_count=<n>, snapshot_date=<date>.

If any step fails, the existing filter remains loaded and the Outpost continues operating.

k-anonymity secondary check

When CREDINT_KANON_ENABLED=true, bloom filter hits trigger an optional confirmation step using the HIBP k-anonymity API:

1. Compute SHA-1 of the candidate credential.
2. Send the first 5 characters of the hex-encoded SHA-1 as a prefix query to CREDINT_KANON_URL.
3. If the full SHA-1 suffix appears in the response, confirm the hit; otherwise treat as a bloom false positive.

k-anonymity frequency buckets (Outpost mode):

Bucket	HIBP count threshold
very_high	> 100,000
high	≥ 10,000
medium	≥ 1,000
low	> 0

k-anonymity is disabled by default. It adds one network round-trip per bloom hit and requires outbound HTTPS access to the HIBP API. Do not enable in air-gap deployments.

DLP pipeline integration

CredInt runs as Tier 4 in the Outpost DLP pipeline, after regex pattern matching (Tier 1), NER entity extraction (Tier 2), and DeBERTa semantic classification (Tier 3). It only receives tokens that earlier tiers have flagged as credential-shaped, which is why the Outpost filter uses a higher FPR target (10%) than the platform filter (0.1%) — the pre-filter upstream substantially reduces the false positive impact.

When a bloom filter match is confirmed, the Outpost emits an entity with:

{
  "entity_type": "compromised_credential",
  "sha1_prefix": "5baa61e4",
  "frequency_bucket": "high",
  "tier": 4
}

The configured policy action (BLOCK or REDACT) is applied based on this entity.

Admin endpoints

Method	Path	Description
GET	/admin/api/credint/status	Current filter load status, entry count, snapshot date, and FPR target.
POST	/admin/api/credint/reload	Trigger an immediate CDN fetch or reload from CREDINT_BLOOM_PATH without waiting for the refresh interval.
POST	/admin/api/credint/kanon	Toggle k-anonymity secondary check at runtime. Body: {"enabled": true}.

GET /admin/api/credint/status example:

curl -s -H "Authorization: Bearer <admin_token>" \
  http://outpost:8080/admin/api/credint/status

{
  "enabled": true,
  "loaded": true,
  "entry_count": 861000000,
  "loaded_at": "2026-03-14T08:22:11Z",
  "file_size_bytes": 556000000,
  "snapshot_date": "2025-11-01",
  "fpr_target": 0.10
}

---

HIBP corpus management

Build script

The platform corpus is built from a raw HIBP SHA-1 ordered-by-prevalence dump using scripts/build_corpus.py. The build environment must have the ARBITEX_AI_MODELS_PATH environment variable set.

export ARBITEX_AI_MODELS_PATH=/data/models

python scripts/build_corpus.py \
  --input /data/hibp/pwned-passwords-sha1-ordered-by-count-v8.txt \
  --top-n 100000000 \
  --processes 125

If the input file is already sorted by count descending (standard HIBP download), pass --presorted to skip the sort phase:

python scripts/build_corpus.py \
  --input /data/hibp/pwned-passwords-sha1-ordered-by-count-v8.txt \
  --top-n 100000000 \
  --processes 125 \
  --presorted

CLI options

Option	Default	Description
--input PATH	—	Path to the HIBP SHA-1 text file (required).
--top-n N	100000000	Number of top-frequency entries to include in the hot cache.
--processes N	125	Number of parallel worker processes for bloom filter construction.
--presorted	false	Skip sort phase; assumes input is already sorted by count descending.

Build phases

Phase 1 — Hot cache build (sequential, ~1 minute): Reads the top --top-n lines of the sorted input file and builds a Python dict mapping SHA-1 → count. Serializes as hot_cache.pkl.

Phase 2 — Bloom filter build (parallel, shared memory): Allocates a shared memory bit array for the full bloom filter. Worker processes consume the remaining SHA-1 entries in batches and write bit positions using lock-free atomic bit operations. Each worker uses mmh3 hash seeds 0 through 9 (k=10 hash functions) to compute bit positions. Serializes the completed filter as corpus.bloom.

Outputs:

File	Description
hot_cache.pkl	Top-N SHA-1 → count dictionary, pickled.
corpus.bloom	Full FastBloomFilter (numpy bit array, mmh3 seeds), pickled.
corpus-meta.json	Snapshot date, item count, FPR target, hot cache size, build duration.

Memory requirements

Phase	Memory
Hot cache (100M entries)	6–10 GB
Bloom filter bit array	~1.54 GB
Peak (both phases overlap briefly)	10–14 GB

Run the build on a host with at least 16 GB of free RAM. The build process does not require GPU resources.

---

Bloom filter sizing and false positive rate

Platform filter (corpus.bloom)

Parameter	Value
Items indexed (n)	857,000,000
FPR target (p)	0.1% (0.001)
Bit array size (m)	~12.3 billion bits (~1.54 GB)
Hash functions (k)	10 (mmh3 seeds 0–9)
Hash algorithm	MurmurHash3 (mmh3)

Outpost filter (ARBF)

Parameter	Value
Items indexed (n)	Large-scale breach corpus
FPR target (p)	10% (0.10) — default CDN filter
Approximate size (uncompressed)	~530 MB
Hash derivation	Double hashing from SHA-256

The Outpost uses a 10% FPR filter by default because Tier 1–3 DLP stages pre-filter the input to credential-shaped tokens only. The effective false positive rate seen in practice is substantially lower than 10%.

Sizing formula

For a bloom filter covering n items at false positive rate p:

m = ceil(-n * ln(p) / (ln 2)^2)   # bit array size
k = ceil((m / n) * ln 2)           # number of hash functions

Available filter sizes (Outpost)

FPR	Uncompressed size	Use case
1%	~1.24 GB	Lower false positive rate; use when Tier 1–3 pre-filtering is disabled
5%	~0.71 GB	Balanced size and accuracy
10%	~0.53 GB	Default; suitable with full DLP pipeline enabled

To use an alternative filter, point CREDINT_CDN_URL to the appropriate CDN path for the desired FPR variant. Contact Arbitex support for the 1% and 5% filter CDN URLs.

---

Health monitoring

Platform microservice health

Poll GET /health on the CredInt container (port 8202) for operational status:

curl -s http://credint:8202/health | jq .

Field	Type	Description
status	string	ok, degraded, or disabled
credint_enabled	boolean	Reflects CREDINT_ENABLED environment variable
corpus_loaded	boolean	true when both hot_cache.pkl and corpus.bloom are loaded
hot_cache_size	integer	Number of entries in the hot cache dict
filter_type	string	FastBloomFilter when bloom is loaded, none otherwise
snapshot_date	string	ISO 8601 date from corpus-meta.json

Recommended monitoring:

• Alert if status transitions from ok to degraded or disabled in production.
• Alert if corpus_loaded: false more than 5 minutes after container start.
• Track hot_cache_size across deployments to detect truncated corpus builds.

Outpost CredInt health

CredInt-specific status:

curl -s -H "Authorization: Bearer <admin_token>" \
  http://outpost:8080/admin/api/credint/status | \
  jq '{enabled, loaded, entry_count, snapshot_date}'

Overall Outpost health (includes DLP tier list):

curl -s http://outpost:8080/health | jq '.dlp.tiers_active'

When CredInt is loaded and active, tiers_active includes "credint":

["regex", "ner", "deberta", "credint"]

If CredInt failed to load (missing filter, ARBF parse error), "credint" is absent from the list while other tiers continue operating.

Recommended monitoring checks

Check	Endpoint	Alert condition
Platform service up	GET :8202/health	status != "ok"
Corpus loaded	GET :8202/health	corpus_loaded: false
Outpost filter loaded	GET /admin/api/credint/status	loaded: false
Outpost filter age	GET /admin/api/credint/status	snapshot_date older than 90 days
DLP tier active	GET /health	"credint" absent from tiers_active

---

Air-gap deployment

In air-gap environments, disable CDN refresh and mount a pre-distributed filter file directly into the Outpost container.

Configuration

docker run -d \
  --name outpost \
  -e OUTPOST_AIRGAP=true \
  -e CREDINT_ENABLED=true \
  -e CREDINT_BLOOM_PATH=/app/credint/credint.bf \
  -e CREDINT_CDN_URL="" \
  -v /path/to/credint.bf:/app/credint/credint.bf:ro \
  arbitex/outpost:latest

Setting CREDINT_CDN_URL to an empty string disables the refresh worker entirely. Setting OUTPOST_AIRGAP=true additionally suppresses startup warnings about missing CDN configuration.

Verification

After container start, confirm the filter is loaded:

curl -s -H "Authorization: Bearer <admin_token>" \
  http://outpost:8080/admin/api/credint/status | \
  jq '{loaded, entry_count, snapshot_date}'

Expected output:

{
  "loaded": true,
  "entry_count": 861000000,
  "snapshot_date": "2025-11-01"
}

Filter distribution

ARBF filter files are distributed in two ways:

• Arbitex release bundles: Each major Outpost release includes a current filter file. Extract credint.bf from the release archive.
• Arbitex support: Request a current filter file through the Arbitex support portal. Provide your snapshot date requirements (minimum required date for compliance purposes).

Staleness management

The snapshot_date field in the ARBF header indicates the corpus cutoff — credentials added to breach databases after that date are not covered. Quarterly filter refresh is recommended for air-gap deployments:

1. Obtain a new credint.bf from Arbitex.
2. Replace the file at the volume mount path on the host.
3. Trigger an in-process reload without restarting the container:

curl -s -X POST \
  -H "Authorization: Bearer <admin_token>" \
  http://outpost:8080/admin/api/credint/reload

4. Verify snapshot_date reflects the new filter.

---

Credential extraction

CredInt operates on candidates extracted from prompt text by the Outpost DLP pipeline before Tier 4 is reached. The extraction stage uses four pattern families:

Pattern family	Detection method
Explicit assignment	Regex: key=VALUE, token: VALUE, secret=VALUE and similar assignment patterns
Environment variables	Regex: VARNAME=VALUE in shell-syntax blocks
Authorization headers	Regex: Authorization: Bearer TOKEN, X-API-Key: TOKEN
High-entropy tokens	Shannon entropy > 3.5 bits per character over a minimum token length

For each candidate:

1. The cleartext value is SHA-1 hashed immediately.
2. The cleartext is discarded from memory.
3. Only the SHA-1 hash is passed to the bloom filter lookup.
4. If a hit is confirmed, the first 8 hexadecimal characters (sha1_prefix) are recorded in the audit log.

The 6 audit columns added by migration 041 are:

• credint_hit — boolean, whether a CredInt match was found
• credint_sha1_prefix — first 8 hex chars of the matched SHA-1
• credint_frequency_bucket — bucket label (critical, high, medium, low)
• credint_tier — always 4 for Outpost, 0 for platform-side checks
• credint_kanon_confirmed — boolean, whether k-anonymity confirmed the hit (null when k-anon disabled)
• credint_action — policy action applied (BLOCK, REDACT, or ALLOW)

No credential cleartext is stored at any point in this pipeline.

---

Routing logic and org sensitivity

CredInt’s enforcement action depends on both the frequency bucket and the org’s DLP sensitivity setting.

Frequency bucket	DLP sensitivity	Action	Audit flag
critical or high	high	soft_block — request blocked	true
critical or high	standard	pass — elevated flag only	true
medium or low	any	pass — flag only	true
No hit	any	pass — no flag	false

soft_block means the AI request is blocked before reaching the model. The user sees a standard block message.

The routing path is recorded in the audit log as credint_routing_path:

Path	Meaning
credint_no_hit	Credential not found in breach corpus
credint_soft_block_high_sensitivity	Critical/high breach hit + high sensitivity = block
credint_elevated_flag_standard	Critical/high breach hit + standard sensitivity = flag
credint_medium_low_flag	Medium/low breach hit = flag only

Per-org sensitivity configuration

CredInt is disabled by default at the org level. Enable it and set the sensitivity via the platform admin API:

# Enable CredInt and set sensitivity
curl -X PATCH https://your-platform/api/v1/admin/org/dlp-config \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"credint_enabled": true, "dlp_sensitivity": "high"}'

Valid values for dlp_sensitivity: standard (default — flags but does not block), high (blocks on critical/high hits).

---

CDN refresh schedule (Outpost)

The platform publishes new bloom filter snapshots when a new batch of credentials is ingested. By default the Outpost checks for updates every 6 hours (CREDINT_REFRESH_INTERVAL_SECONDS=21600). The refresh is a conditional GET — if the filter has not changed since the last download (HTTP 304), no update is applied.

To force an immediate refresh without restarting the Outpost:

curl -s -X POST \
  -H "Authorization: Bearer <admin_token>" \
  http://outpost:8080/admin/api/credint/reload

The endpoint returns the new filter metadata (snapshot timestamp, entry count, FPR target) on success.

---

Outpost admin status panel

The Outpost admin interface exposes a Credential Intelligence status panel at GET /admin/credint/status (or equivalently GET /admin/api/credint/status):

{
  "enabled": true,
  "mode": "cdn",
  "filter_loaded": true,
  "filter_snapshot_at": "2026-03-12T08:00:00Z",
  "filter_entry_count": 4200000,
  "filter_fp_rate": 0.001,
  "last_refresh_at": "2026-03-12T08:00:04Z",
  "last_refresh_status": "ok",
  "hits_last_24h": 3,
  "policy_action": "BLOCK"
}

Field	Description
filter_loaded	Whether a filter is currently active in memory
filter_snapshot_at	When the active filter snapshot was generated
filter_entry_count	Number of credential hashes in the filter
filter_fp_rate	Configured false positive rate
last_refresh_at	Timestamp of the most recent filter download attempt
last_refresh_status	"ok", "unchanged", or "error"
hits_last_24h	Credential matches detected in the last 24 hours

---

False positive handling (Outpost)

At the default 0.1% false positive rate, roughly 1 in 1000 credential-like strings will produce a spurious hit. If false positives are causing production disruption:

1. Switch to REDACT action instead of BLOCK — the value is masked but the request proceeds.
2. Report the false positive via the admin panel using the POST /admin/credint/report-fp endpoint, which submits the hash of the matched value to the platform for exclusion in the next filter snapshot. False positive reports are applied to the next filter generation cycle, typically within 24 hours.
3. Request a lower-FPR filter — contact Arbitex support to request a 1% or 5% FPR filter build for your deployment (see Available filter sizes above).