Platform test failure inventory

This document catalogs the platform’s known test failures, originally identified as AT-003 during the audit cycle. The initial count was 182 failures; platform sprint platform-0034 (T490) fixed 12, leaving 170 remaining.

Current test baseline: 7,384 passing / 170 failing.

For the raw test-level catalog, see arbitex-platform/backend/docs/test_skips.md.

Failure categories

Category	Count	Root cause	Resolution criteria
mTLS cert required	~27	Tests require a valid client certificate signed by the platform CA	Set `CLOUD_CA_CERT_PATH` to a valid CA bundle in the test environment
Bloom filter startup gate	~20	Bloom filter must be initialized before request processing; tests hit 503	Pre-warm bloom filter in test setup or mock the readiness gate
Missing optional packages	~20	DLP file parsers depend on optional Python packages	`pip install python-docx python-pptx openpyxl striprtf`
Code behavioral drift	~15	Test assertions don’t match current implementation (HSTS headers, HMAC format, entity counts)	Update test assertions to match current behavior
PostgreSQL required	~12	Tests use Alembic migrations or `inet` column types unavailable in SQLite	Run against PostgreSQL (`DATABASE_URL=postgresql://...`)
DLP microservice unavailable	~9	NER endpoint not running in CI	Start `DLP_NER_ENDPOINT` on port 8200 or mock responses
Patch target mismatch	~5	`unittest.mock.patch` paths don’t match refactored module layout	Update patch targets to correct import paths
Redis required	~8	Tests need a live Redis instance for rate limiting / caching	Set `REDIS_URL` (e.g., `redis://localhost:6379/1`)
API model field changes	~6	Response schemas changed but test fixtures weren’t updated	Update test fixtures to match current Pydantic models
Azure / HashiCorp Vault	~6	Tests require `AZURE_KEYVAULT_URL` or `VAULT_ADDR`	Provide vault credentials or mock the vault client
DeBERTa service unavailable	~3	Tier-3 classifier endpoint not running	Start DeBERTa validator on port 8201 or mock

Total: ~170 (sum may vary ±5 due to overlap between categories).

Category details

mTLS cert required (~27 failures)

Affected test files: test_mtls_auth.py, test_cloud_sync.py, test_outpost_audit_sync.py, test_internal_api.py

Root cause: The platform enforces mutual TLS on internal endpoints (/v1/internal/*). Tests that exercise these endpoints fail with 403 Certificate Required when no client cert is provided.

Environment setup:

# Generate test CA and client cert
openssl req -x509 -newkey rsa:4096 -keyout ca-key.pem -out ca-cert.pem \
  -days 365 -nodes -subj "/CN=Arbitex Test CA"

openssl req -newkey rsa:4096 -keyout client-key.pem -out client-csr.pem \
  -nodes -subj "/CN=test-client"

openssl x509 -req -in client-csr.pem -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -out client-cert.pem -days 365

export CLOUD_CA_CERT_PATH=./ca-cert.pem
export CLIENT_CERT_PATH=./client-cert.pem
export CLIENT_KEY_PATH=./client-key.pem

Resolution status: 14 of the original ~41 mTLS tests were fixed in platform-0034 (T490) by adding _verify_issuer() to middleware/mtls.py. The remaining ~27 need the test environment configured with valid certificates.

Bloom filter startup gate (~20 failures)

Affected test files: test_request_dedup.py, test_middleware_chain.py, test_health.py

Root cause: The bloom filter middleware returns 503 until initialization completes. In tests that don’t pre-warm, the first requests fail.

Resolution: Either pre-warm the bloom filter in test setup:

@pytest.fixture(autouse=True)
def warm_bloom(app):
    app.state.bloom_filter.initialize(expected_items=1000, fp_rate=0.01)

Or mock the readiness check:

@pytest.fixture(autouse=True)
def skip_bloom_gate(monkeypatch):
    monkeypatch.setattr("middleware.bloom.is_ready", lambda: True)

Missing optional packages (~20 failures)

Affected test files: test_dlp_docx.py, test_dlp_pptx.py, test_dlp_xlsx.py, test_dlp_rtf.py

Root cause: DLP file content extraction uses optional parsers. Tests import these directly and fail with ModuleNotFoundError.

Resolution:

pip install python-docx python-pptx openpyxl striprtf

These are listed in requirements-dev.txt but not in the base requirements.txt to keep the production image slim.

PostgreSQL required (~12 failures)

Affected test files: test_alembic_*.py, test_inet_columns.py, test_migration_*.py

Root cause: SQLite (used in CI by default) doesn’t support PostgreSQL-specific features: inet column type, Alembic migration chain execution, advisory locks.

Resolution: Run these tests against a PostgreSQL instance:

export DATABASE_URL="postgresql://test:test@localhost:5432/arbitex_test"
pytest tests/database/ -k "alembic or inet or migration"

DLP microservice unavailable (~9 failures)

Affected test files: test_dlp_ner.py, test_dlp_pipeline.py, test_dlp_integration.py

Root cause: The DLP NER microservice must be running on DLP_NER_ENDPOINT (default port 8200). Without it, tests fail with connection refused.

Resolution: Start the NER service or mock:

# Start the service
DLP_NER_ENDPOINT=http://localhost:8200 python -m dlp_ner.server &

# Or mock in tests
@pytest.fixture
def mock_ner(monkeypatch):
    monkeypatch.setenv("DLP_NER_ENDPOINT", "http://mock:8200")
    # ... mock HTTP responses

Redis required (~8 failures)

Affected test files: test_rate_limit.py, test_cache.py, test_session.py

Root cause: Rate limiting and caching tests need a live Redis connection.

Resolution:

export REDIS_URL="redis://localhost:6379/1"
pytest tests/ -k "rate_limit or cache or session"

Other categories

Code behavioral drift (~15): Scattered across test files. Each test needs its assertion updated to match current behavior. The platform team tracks these in backend/docs/test_skips.md with specific line references.

Patch target mismatch (~5): After module refactoring, mock.patch("old.module.path.func") calls need updating. The correct paths are documented in test_skips.md.

API model field changes (~6): Pydantic model updates (field renames, new required fields) broke fixture-based tests. Update test fixtures to match current schema.

Azure / HashiCorp Vault (~6): Require either real vault credentials or mocked vault clients. Set AZURE_KEYVAULT_URL or VAULT_ADDR.

DeBERTa service (~3): Tier-3 classifier tests need the DeBERTa service on port 8201. Mock or run the service.

CI environment matrix

To run the full test suite with all failures resolved, the CI environment needs:

Dependency	Configuration
PostgreSQL 15+	`DATABASE_URL=postgresql://...`
Redis 7+	`REDIS_URL=redis://...`
mTLS certificates	`CLOUD_CA_CERT_PATH`, `CLIENT_CERT_PATH`, `CLIENT_KEY_PATH`
DLP NER service	`DLP_NER_ENDPOINT=http://localhost:8200`
DeBERTa validator	Port 8201 running
Python optional packages	`python-docx python-pptx openpyxl striprtf`
Azure Key Vault (or mock)	`AZURE_KEYVAULT_URL`
HashiCorp Vault (or mock)	`VAULT_ADDR`

Tracking

Metric	Value
Original failure count (AT-003)	182
Fixed in platform-0034 (T490)	12
Remaining	170
Estimated environment-only fixes	~95 (mTLS + bloom + packages + PostgreSQL + Redis + vault + DLP + DeBERTa)
Estimated code-change fixes	~75 (drift + patch targets + model fields)