Entity extraction regex over-matching — co-occurrence graph crowding from noisy default patterns

[codenamekt]

Summary

The entity extractor's default regex patterns for hostname and docker_image are too
broad. They tag common text patterns (hyphenated tokens, bare word:word) as entities,
which will produce a high rate of false positives in the co-occurrence graph as the
memory store accumulates entries at volume. A noisier graph means lower signal for
entity-based recall (entity_graph and graph_walk features).

Evidence

The current patterns from hexus/entity_extractor.py:

DEFAULT_PATTERNS = {
    "url": r'https?://[^\s<>"]+',
    "domain": r"\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b",
    "hostname": r"\b(?:[a-zA-Z][a-zA-Z0-9-]{0,61}[a-zA-Z0-9])\b",
    "docker_image": r"\b[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+(?::[a-zA-Z0-9_.-]+)?\b|(?<![a-zA-Z0-9_.-])\b[a-zA-Z0-9_.-]+:[a-zA-Z0-9_.-]+\b",
    "ip_address": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",
    "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b",
    "version_tag": r"\bv?\d+\.\d+(?:\.\d+)?(?:-[a-zA-Z0-9.]+)?\b",
}

The hostname pattern (line 9)

r"\b(?:[a-zA-Z][a-zA-Z0-9-]{0,61}[a-zA-Z0-9])\b"

This matches any hyphenated token that starts with a letter and ends with
alphanumeric. Real-world false positives from a real memory store:

Text in a memory entry	Matches hostname?
all-MiniLM-L6-v2	Yes — three times (all, MiniLM, L6)… actually after re-reading the pattern, since it requires [a-zA-Z] start and [a-zA-Z0-9] end, all (ends in 'l' ✓) matches. MiniLM (ends in 'M' ✓) matches. L6 (ends in '6' ✓) matches. Actually all-MiniLM-L6-v2 split by non-word chars won't match the full thing but each segment that fits the pattern does.
pre-trained	Yes
end-to-end	Yes
high-level	Yes

The docker_image pattern (line 11)

r"\b[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+(?::[a-zA-Z0-9_.-]+)?\b|(?<![a-zA-Z0-9_.-])\b[a-zA-Z0-9_.-]+:[a-zA-Z0-9_.-]+\b"

The second alternative (after |) matches any bare word:word pattern with no /:

Text	Matches docker_image?
key:value	Yes
config:enable	Yes
model:gpt-4	Yes
result:ok	Yes
MiniLM-L6-v2:latest	Yes (if a user writes this, it tags as docker image)

The domain pattern (line 8)

r"\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b"

This is reasonable for real domains but also catches bare TLD-level patterns in
prose like "I work at X.Y" if X is a single char. Minor issue.

Impact on the co-occurrence graph

The entity graph and graph_walk features count co-occurrences between entity types.
Every time a memory entry contains "all-MiniLM-L6-v2", the entity extractor tags
all, MiniLM, L6 as hostnames. These then co-occur with whatever the actual
topic of the entry is (e.g., "embedding", "recall", "pgvector"). After enough entries,
the graph becomes crowded with false edges that dilute real associations. The entity
feature is noisier than it needs to be, reducing the value of entity-based recall.

The HEXUS_ENTITY_EXTRACTOR_ENABLED and HEXUS_ENTITY_EXTRACTOR_PATTERNS env vars
exist as escape hatches (store.py:111-120), but the defaults are the first thing
new users hit and the noisy patterns affect everyone who doesn't configure overrides.

Proposed approach

Narrow the default patterns to reduce false positives without losing real entities:

1. hostname — Require at least one dot (.), matching actual DNS-style names:
   r"\b[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+\.[a-zA-Z]{2,}\b"
   This drops bare single-label tokens like all, MiniLM, pre-trained.

2. docker_image — Drop the bare word:word alternative (the second branch).
   Always require a /:
   r"\b[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+(?::[a-zA-Z0-9_.-]+)?\b"
   This drops key:value, model:gpt-4, result:ok but still matches
   postgres:16-alpine, python:3.11-slim.

3. Add a cap on entities per entry — e.g., max 50 entities extracted from a
   single text. Prevents a pathological text from generating thousands of entities
   and overwhelming the graph in one write.

4. Add entity-count logging — emit a logger.info with the entity type and count
   per extraction call, so there's visibility into how noisy the extractor is in
   production.

5. Update tests — tests/test_smoke.py contains entity extraction tests
   (functions test_entity_tagging_and_graph, test_append_turn_entity_extraction).
   Add assertions that common false-positive patterns (all-MiniLM-L6-v2,
   key:value, pre-trained) do not match after the regex changes.

Open questions

• Should version_tag also be tightened? It currently matches semver-like strings
  which are mostly fine, but v1.2 in prose like "we upgraded to v1.2" is accurate
  tagging — no change needed IMO.
• The entity cap (item 3 above): should truncated entries be logged, or just silently
  capped? Logging once per capped entry is probably right for Phase 1.