CredInt Extractor — Accuracy & Testing

The Credential Intelligence (CredInt) pipeline operates in two stages: extraction (finding candidate credentials in text) and lookup (checking those candidates against the breach corpus). This page covers the accuracy validation framework for the extraction stage — how the extractor is tested, what its error modes are, and how the test suite handles credential-like test fixtures.

For the lookup stage architecture and breach corpus details, see Credential Intelligence.

---

What the extractor does

The credential extractor (credential_extractor.py) runs inline in the DLP pipeline hot path. It scans conversation text for credential-like strings and returns a list of CredentialCandidate objects. Each candidate includes:

• value — the raw plaintext credential candidate (must be hashed for lookup and never logged)
• context_type — which pattern family produced this candidate
• position — character offset in the original text
• raw_match — the full matched string for debugging only

The extractor is synchronous, has no database or network dependencies, and returns candidates in position order with duplicates removed.

Pattern families

Four pattern families are applied in sequence:

Family	What it matches	Example
explicit_assignment	Variable-name patterns with = or : delimiters	password=s3cr3t, api_key: sk-abc123
environment_variable	Shell/Docker env vars whose names contain credential keywords	DB_PASSWORD=changeme, export AWS_SECRET_ACCESS_KEY=...
authorization_header	HTTP Authorization and X-API-Key headers	Authorization: Bearer eyJhb..., X-API-Key: abc123
high_entropy_code	High-entropy strings in code-like text regions	Quoted or bare tokens with Shannon entropy > 3.5 bits/char

After all families run, candidates are deduplicated by (value, position) and sorted by position.

---

How accuracy is measured

Extraction accuracy vs. lookup accuracy

The accuracy framework distinguishes two separable concerns:

1. Extraction accuracy — Does the extractor find the credentials that are present? Does it avoid firing on non-credentials?
2. Lookup accuracy — When a candidate is found, does the breach corpus correctly classify it as compromised or not?

These are measured independently. Extraction accuracy is covered by the extractor unit test suite. Lookup accuracy is governed by the CredInt microservice’s Bloom filter false positive rate (0.1% by design).

Extraction precision and recall

The extractor test suite (test_credential_extractor.py) covers both success cases (true positives) and exclusion cases (true negatives):

True-positive tests — Verify that specific credential formats are extracted. Every supported pattern has explicit tests:

# Explicit assignment family
results = _extract_explicit_assignment("password=mysecret123")
assert results[0].value == "mysecret123"

# Environment variable family
results = _extract_environment_variable("export DB_PASSWORD=changeme")
assert any("changeme" in c.value for c in results)

# Authorization header family
results = _extract_authorization_header("Authorization: Bearer eyJhb")
assert results[0].context_type == CredentialContextType.AUTHORIZATION_HEADER

# High-entropy token in code context
results = _extract_high_entropy_code('token = "aB3!xQ9#mLpR7$kT2"')
assert len(results) >= 1

True-negative tests — Verify that common non-credentials are NOT extracted:

# Prose password mentions should not fire
results = _extract_explicit_assignment("Enter your password below")
assert len(results) == 0

# Pure integer values are not credentials
results = _extract_explicit_assignment("password=12345678")
assert len(results) == 0

# Short values (< 6 chars) are excluded
results = _extract_explicit_assignment("password=abc")
assert len(results) == 0

# Non-credential variable names are ignored
results = _extract_explicit_assignment("word=hello")
assert len(results) == 0

Running the extractor accuracy suite

# Full extractor unit tests
PYTHONPATH=. pytest backend/tests/services/test_credential_extractor.py -v

# Focus on a specific pattern family
PYTHONPATH=. pytest backend/tests/services/test_credential_extractor.py -k "explicit" -v
PYTHONPATH=. pytest backend/tests/services/test_credential_extractor.py -k "entropy" -v

---

The common-word exclusion list

What it is

The _COMMON_WORDS frozenset in credential_extractor.py is a deny-list of strings that match the structural criteria for high-entropy candidates (length, character class mix) but are known to be non-credentials. It is applied exclusively to the high_entropy_code pattern family — the only family that lacks an explicit structural signal (like an explicit key name or HTTP header) and therefore relies on entropy thresholds alone.

Current entries:

_COMMON_WORDS = frozenset({
    "localhost", "password", "username", "hostname", "database",
    "example", "default", "template", "placeholder", "changeme",
    "undefined", "null", "none", "true", "false", "integer", "string",
})

How it works

The check occurs inside _is_high_entropy_candidate(). After passing length, UUID, URL, and entropy checks, the candidate’s lowercased value is compared against _COMMON_WORDS. If a match is found, the candidate is rejected:

if value.lower() in _COMMON_WORDS:
    return False  # Common word, not a credential

The check is case-insensitive. "PASSWORD", "Password", and "password" are all excluded.

Scope and limitations

The common-word list is intentionally minimal. Entropy thresholds do the heavy lifting — the list only covers words that:

1. Appear frequently as placeholder values in code and configuration
2. Have enough character diversity to pass entropy checks despite being common

The list does not attempt to be exhaustive. Operational false positives from high-entropy code detection are better addressed by:

• Improving entropy threshold tuning
• Relying on Tier 2 and Tier 3 DLP pipeline stages to suppress false-positive classifications downstream
• Using the CredInt lookup result itself — genuine credentials in the breach corpus will return hit=true; placeholder strings like “changeme” should not

Adding to the list

Additions to _COMMON_WORDS should be conservative. Broad additions reduce recall on genuine credentials. A candidate word should only be added if:

1. It appears in real codebases as a non-credential string
2. Its character composition causes it to pass the entropy and character-class checks
3. The addition is tested against the extractor corpus to confirm no recall regression

Changes to _COMMON_WORDS require a passing run of the full extractor test suite before merge.

---

High-entropy token quality gates

The high_entropy_code family applies layered filtering to minimize false positives:

Gate	Threshold	Rationale
Length	8–128 characters	Too short = not a real credential; too long = not a typical credential value
Shannon entropy	> 3.5 bits/character	Real credentials have high randomness; common words and repeated patterns do not
Character class diversity	≥ 2 distinct classes	Passwords and API keys use mixed character sets; pure-lowercase tokens are less likely
UUID exclusion	Regex match	UUIDs are structural identifiers, not credentials
URL exclusion	Prefix match	HTTP/HTTPS URLs are not credentials
All-digit exclusion	isdigit() check	Pure numbers are not credentials
Code context	Line contains =, :, {, }, ;, ", ', `	High-entropy tokens in prose are less reliable than tokens in code-like regions

A candidate must pass all gates. Any single failure suppresses extraction.

---

Known test credential handling

The test infrastructure problem

The extractor test suite uses realistic credential-shaped strings (e.g., "hunter2x!", "aB3!xQ9#mLpR7$kT", "sk-abc123def456xyz") to verify that specific formats are extracted. These strings are deliberately credential-like in structure — that is the point of the test.

Some of these test strings may also exist in real breach corpora. This creates a testing concern: when the test suite sends a test string to the CredInt microservice for integration testing, a lookup hit could indicate either a genuine breach match or a coincidental corpus overlap with a known test fixture.

How the test suite handles this

The integration tests for the CredInt client (test_credint_integration.py) use an in-process mock transport that intercepts all HTTP calls. No requests reach the live CredInt microservice during unit or integration tests:

class _MockTransport(httpx.AsyncBaseTransport):
    """In-process transport that intercepts CredInt HTTP calls."""

    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
        key = (request.method, request.url.path)
        status, body = self._responses.get(key, (500, {"error": "not configured"}))
        return httpx.Response(status_code=status, json=body)

This means:

• Test credential strings never reach the breach corpus during test runs.
• Test results are entirely determined by the mock response configuration.
• The CredInt microservice itself requires no test-mode flag or corpus exclusions.

Lookup result semantics

When a CredInt hit occurs in production, the sha1_prefix field in the audit log records the first 8 hex characters of the SHA-1 hash of the candidate value. This prefix:

• Cannot be reversed to recover the original credential
• Can be used for cross-request correlation (if the same credential appears in multiple requests, the same prefix appears in each audit log entry)
• Is NOT a match indicator — the prefix is logged regardless of whether the lookup returned hit=true or hit=false

Test credentials used in QA environments should be rotated before use in production. If a test credential appears in breach data (i.e., it was a real credential used by someone who was breached), its sha1_prefix will match in the live corpus. The appropriate response is to treat the finding as a genuine signal: the test credential should be changed.

---

Accuracy thresholds and CI gates

The extractor test suite serves as a functional gate. CI enforces:

Check	Requirement
All extractor unit tests pass	Required — no exceptions
All pattern families have at least one TP and one TN test	Required
_COMMON_WORDS changes include regression test update	Required
Integration tests with mock transport pass	Required

Unlike the DLP accuracy harness, there are no numeric F1 thresholds for the extractor. The extractor is pattern-based and its behavior is fully deterministic — if a pattern-based test passes, the extractor is working correctly for that case. Coverage completeness is enforced by the test authors, not by threshold metrics.

---

False positive guidance

Explicit assignment and environment variable families

False positives from these families require an explicit keyword match (e.g., password, token, secret, api_key) at a word boundary. False positives in prose (“enter your password below”) are suppressed by the word-boundary requirement in the regex. If a false positive occurs, check:

1. Is the triggering keyword at a word boundary in the text?
2. Is the value (after the = or :) at least 6 characters and not purely digits?

If both are true and the candidate is genuinely not a credential, it may indicate a text pattern that warrants a test case addition and pattern refinement.

High-entropy code family

This family has the highest false-positive risk because it relies on entropy heuristics rather than explicit key names. Common false-positive sources:

• Base64-encoded data (configuration values, UUIDs in unusual formats)
• Hash strings in test output
• Minified code tokens

High-entropy false positives that reach the CredInt lookup will typically return hit=false — meaning they are not in the breach corpus. The system treats a non-hit as no action. The operational impact is limited to unnecessary CredInt lookups and the associated audit log entries.

---

See also

• Credential Intelligence — full pipeline architecture including the breach corpus, lookup protocol, and audit log fields
• DLP Accuracy — Methodology & Results — precision/recall methodology for the DLP regex pipeline
• Credential Intelligence Admin — configuration, health monitoring, and audit log field reference