Content Category Accuracy — Methodology & Validation

Content category classification uses a keyword-based L1 classifier that performs case-insensitive substring matching against the full text of each conversation turn. Because classification is deterministic and fully observable, it can be validated with precision — the keyword harness measures exactly which categories the classifier fires on, how often it fires correctly, and how often it fires incorrectly.

This page describes the validation methodology for the content category classifier. The parallel methodology for the DLP regex pipeline is documented in DLP Accuracy — Methodology & Results.

---

How classification works

The classifier is documented in full in Content Categories. In summary:

1. Each conversation turn is lowercased once.
2. Every enabled category’s keyword list is checked via case-insensitive substring matching.
3. The first keyword match for a category fires that category; remaining keywords for the same category are skipped.
4. When a child slug fires (e.g., legal.contracts), the parent domain (legal) is automatically propagated.
5. Output is a deduplicated, sorted list of matched category slugs.

For accuracy purposes the important property is that classification is entirely determined by the keyword lists. There are no probabilistic weights, no model thresholds to tune. Every false positive and false negative traces directly back to a specific keyword or its absence.

---

Accuracy harness design

What the harness measures

The harness measures per-category precision and recall across a labeled corpus of text samples. Each sample specifies:

• The input text.
• Which category slugs should fire (expected_matches).
• Which category slugs should NOT fire (expected_negative).

The harness runs the actual classifier against each sample (using the same classify_content code path as production) and counts:

Symbol	Meaning
TP	Category fires and should have fired
FP	Category fires and should NOT have fired
FN	Category should have fired but did not
TN	Category did not fire and should not have fired

From these counts the harness computes precision (TP / (TP + FP)), recall (TP / (TP + FN)), and F1 per category and in aggregate.

Corpus structure

The keyword accuracy corpus is organized by domain and consists of two sample types:

Positive samples — text that contains topic-relevant content and should trigger the target category. A legal contract excerpt should match legal.contracts. A text containing investment strategy discussion should match financial.investment. Positive samples measure recall.

Negative samples — text that could plausibly be confused with a category but should not trigger it. A document discussing legal file formats should not trigger legal.contracts simply because it contains the word “legal”. A document about financial file formats should not trigger financial.accounting. Negative samples measure precision and identify keyword over-reach.

Each sample is a JSON object with the schema:

{
  "id": "legal-contracts-tp-001",
  "text": "The parties agree that this indemnification clause shall...",
  "expected_matches": ["legal", "legal.contracts"],
  "expected_negative": ["financial", "hr"],
  "category": "legal.contracts",
  "difficulty": "easy",
  "notes": "Classic contract language"
}

The expected_matches list includes both the child slug and any parent slugs the harness should expect due to parent propagation. The expected_negative list explicitly names categories that should not fire — this drives precision measurement.

Running the harness

The harness is a standalone pytest-based suite that loads the corpus and runs the classifier in isolation (no database required — keyword lists are loaded from the fixture data):

# Run full accuracy suite
pytest backend/tests/test_content_categories.py -m accuracy -v

# Run a specific domain
pytest backend/tests/test_content_categories.py -m accuracy -k "legal" -v

# Run and emit a JSON report
pytest backend/tests/test_content_categories.py -m accuracy \
  --json-report --json-report-file=accuracy-report.json

To evaluate against a live database keyword configuration (for custom org overrides):

CONTENT_CATEGORY_HARNESS_ORG_ID=<org-uuid> \
pytest backend/tests/test_content_categories.py -m accuracy

---

How to read results

Precision

Precision measures how many classifier fires were correct.

Precision = TP / (TP + FP)

A precision of 1.000 means every time a category fired, it was supposed to fire. A precision of 0.800 means 20% of fires were false positives — the classifier matched text it shouldn’t have.

Low precision on a category means its keyword list is too broad. Common causes:

• Short or common keywords that appear naturally in unrelated text (e.g., keyword "legal" in a category that should only match legal contracts, but fires on “legally blonde” or “legal file format”)
• Keywords that are substrings of longer words (e.g., keyword "hr" matching "three", "chr", "chrome")

Recall

Recall measures how many samples that should have matched, did match.

Recall = TP / (TP + FN)

A recall of 0.900 means the classifier missed 10% of samples it should have caught. Low recall on a category means its keyword list is too sparse — relevant text is not covered.

F1 score

F1 is the harmonic mean of precision and recall, useful when you want a single number. It penalizes both false positives and false negatives equally.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

A perfect F1 of 1.000 means the category has no false positives and no false negatives on the test corpus.

Per-category vs. aggregate

The harness reports metrics at two levels:

Per-category — precision, recall, F1 for each slug independently. Use this to identify which specific categories need keyword list attention.

Aggregate — macro-averaged metrics across all categories. Use this for CI gating and overall health assessment.

---

Negative keywords

What they are

Negative keywords are terms whose presence in a text should suppress category classification, even if a positive keyword also matches. They are the primary mechanism for resolving precision problems without sacrificing recall.

A concrete example: the category medical.diagnoses has the keyword "disorder". Without a negative keyword, a text about “personality disorder not elsewhere classified” triggers the category even when the conversation is about a data classification problem, not a medical condition. Adding "data disorder" as a negative keyword suppresses the match when that phrase is present.

Negative keywords are checked after positive keyword matching. The logic is:

1. Check whether any positive keyword matches the text.
2. If a positive match is found, check whether any negative keyword also matches the text.
3. If a negative keyword matches, suppress the category result.

Managing negative keywords

Negative keywords are stored on the ContentCategoryDefinition row alongside the positive keyword list, in a separate negative_keywords field. They are managed through the same admin API:

# Add a negative keyword to an existing category
PUT /api/v1/admin/content-categories/{slug}
{
  "negative_keywords": ["data disorder", "class disorder", "sort disorder"]
}

When evaluating whether a negative keyword is needed:

1. Identify a false-positive sample in the accuracy corpus.
2. Find which keyword(s) triggered the match.
3. Identify the phrase in the false-positive text that should be suppressed.
4. Add a negative keyword that is specific enough to suppress only that pattern without affecting genuine positives.
5. Run the harness to confirm precision improves and recall does not regress.

Negative keyword caveats

Negative keywords are substring-matched with the same case-insensitive logic as positive keywords. Overly broad negative keywords can suppress legitimate matches. When adding a negative keyword:

• Prefer multi-word phrases over single words.
• Test against the full corpus, not just the false-positive case.
• Use the expected_negative sample mechanism to document the intent.

---

CI quality gates

Gate configuration

The keyword accuracy harness is integrated into the platform CI pipeline as a quality gate. The gate runs automatically on any pull request that modifies:

• ContentCategoryDefinition keyword lists (seeder or migration changes)
• The classify_content function
• Category test corpus files

Gate thresholds are defined in backend/tests/test_content_categories.py:

_CATEGORY_GATE_TARGETS = {
    "precision": 0.90,   # aggregate across all categories
    "recall":    0.85,   # aggregate across all categories
    "f1":        0.87,   # aggregate across all categories
}

A pull request that causes any aggregate metric to fall below its threshold fails CI.

What a gate failure means

Precision failure (< 0.90) — the keyword change introduced new false positives. Look at the failing samples in the test output. The category name and the triggering keyword will be identified. Either the keyword is too broad, or a negative keyword is needed.

Recall failure (< 0.85) — the keyword change caused the classifier to miss samples it previously caught. This can happen when keywords are renamed, narrowed, or when a seeder change removes a keyword. Check which samples are now false negatives and restore the coverage.

F1 failure — typically indicates a trade-off went too far in one direction. Review both precision and recall outputs to identify the category pulling the aggregate down.

Updating the baseline

When keyword list improvements raise the accuracy metrics above prior baselines, update the stored baseline:

# From project root
pytest backend/tests/test_content_categories.py -m accuracy \
  --save-category-baseline

The baseline is stored in backend/tests/baselines/content_categories_current.json. Commit this file alongside the keyword list changes that produced the improvement.

---

Interpreting accuracy by category

Built-in vs. custom categories

Arbitex ships with 8 domain categories and 26 subcategories seeded from seed_content_categories.py. Built-in keyword lists are curated to balance precision and recall across general-purpose enterprise AI usage.

Custom org-scoped categories (added via the admin API without a global row) are not covered by the built-in corpus. Organizations adding custom categories should extend the accuracy corpus with org-specific samples and run the harness with CONTENT_CATEGORY_HARNESS_ORG_ID set.

Expected characteristics

Property	Expected behaviour
Precision on narrow domains (e.g., medical.diagnoses)	Higher — medical vocabulary is distinctive
Precision on broad domains (e.g., legal)	Lower — legal vocabulary appears in many contexts
Recall on explicit slugs (e.g., financial.invoices)	High — invoice-related vocabulary is specific
Recall on parent-only slugs (e.g., legal)	Automatic — recall is inherited from child matches

Parent domain precision depends on child precision. If legal.contracts has a high FP rate, legal will also accumulate false positives from the same samples.

Known precision challenges

The following built-in categories have known precision challenges on adversarial samples and are areas of ongoing keyword list refinement:

Category	Challenge	Mitigation
security.research	“research” is a common word	Requires multi-word keywords or negative suppression of non-security contexts
hr.performance	“performance” triggers in technical contexts (performance benchmarks, system performance)	Negative keywords for "system performance", "benchmark", "latency"
legal (parent)	Inherits FPs from any child with precision issues	Fix child category keyword lists first

---

See also

• Content Categories — classifier architecture, keyword management, policy engine integration
• DLP Accuracy — Methodology & Results — the parallel accuracy framework for the DLP regex pipeline
• Admin API: Content Categories — CRUD endpoints for managing category definitions and keyword lists