Stop embedding .env files; stop returning document text three times

[seiz888]

1. .env files are read and sent to the embedding provider

.env is present in the isIndexable() extension allowlist, so rag_index_project
reads any .env (and any *.env) in a scanned project, sends its contents to the
configured embedding provider — Voyage AI by default — and persists it in the vector
store, where it stays semantically retrievable by any agent with access to the
collection.

This is the documented happy path, not an edge case: the README tells agents to call
rag_index_project to sync file changes, and a typical Node/Next/Django project keeps
a .env at its root.

It looks like an oversight rather than a decision, because the handling is inconsistent:

file	filepath.Ext()	indexed before this PR
.env	.env	yes
prod.env	.env	yes
.env.local	.local	no
.env.production	.production	no
.env.example	.example	no

Nobody deliberately designs "index .env but skip .env.production". The walker also
indexes .gitignore as a document but never consults it, so a .env that is excluded
from version control precisely because it holds credentials is still read.

Fix: drop .env from the allowlist and add an isSensitive() gate after
isIndexable().

Scope, stated honestly

This is not remotely exploitable — it needs an operator to index a directory containing
a .env. The impact is that the operator's own credentials leave their machine to a
third-party API and persist in their own vector store. Moderate severity, high
likelihood, two-line root cause.

Key material was never affected. id_rsa, *.pem, *.key, .netrc and .pgpass
have extensions absent from codeExts, so isIndexable() already rejected them; I
checked rather than assumed. isSensitive() now covers them anyway so the guarantee no
longer depends on codeExts never growing.

What the second gate does newly protect are names that are otherwise perfectly
indexable through allowlisted extensions: secrets.json, db_password.yml,
credentials.yaml. The name-substring match is deliberately broad — skipping a document
that merely mentions a password is a cheaper mistake than embedding a live credential.

2. Retrieval returns the same text three times

The Qdrant read path copies every string payload field into Result.Meta, and content
is a payload field. So each result carried the document text in both .Content and
.Meta["content"]. rag_retrieve_context then returned the concatenated context
string alongside those chunks — putting the same text into one tool response three
times. Agents pay for all three in tokens; two are unreadable noise.

• exclude the content key from the metadata map on read (search and export)
• return chunks from rag_retrieve_context as references (id, score, meta)

Warning

Breaking: rag_retrieve_context no longer returns chunks[].Content. The
context string already carries every chunk's text with its score, so callers should
read it from there. Flagged in the changelog. The repo is pre-1.0, so this seemed
preferable to shipping an opt-in flag that would leave the default path paying triple.

3. Metadata filtering for memory-shaped collections

While in the read path: PointInfo now carries the remaining payload tags in meta,
and rag_list_points accepts an exact-match filter on any metadata key rather than
only source_file. Both are needed when a collection holds agent memory tagged with
bucket/kind/ts instead of a code index — previously those tags could neither be
filtered on nor read back.

Tests

Five new tests. Each was mutation-checked: reverted the fix, confirmed the test fails,
restored it, confirmed it passes.

• TestIsSensitive — the predicate, including the deliberate over-inclusion
• TestIndexerSkipsSensitiveFiles — writes secrets to disk, asserts they never reach the provider
• TestQdrantSearchMetaExcludesContent — text returned once, useful tags survive
• TestQdrantListPointsSurfacesMeta — bucket/kind/ts exposed, promoted fields not echoed

go build ./... and go vet ./... are clean. pkg/rag and pkg/indexer pass.

pkg/httpapi has 14 failures on Windows — verified pre-existing by running the same
suite on a pristine clone of main: identical 14, so none are introduced here. They are
POSIX-permission and HOME-path assumptions (expected 0600 ... got 666).

🤖 Generated with Claude Code