fix(privacy): redact credentials on every write path and mask them when shown - #569
Open
kevintseng wants to merge 6 commits into
Open
kevintseng wants to merge 6 commits into
kevintseng wants to merge 6 commits into
Conversation
…lines Credential-shaped text (a connection string's password, `token=…`, API keys, PEM private keys including truncated and JSON-escaped ones, a `Bearer` token on the next line) is replaced with `***REDACTED***` before it is stored, on every write path: - `remember`, `learn` (including the derived lesson name), `task_state`, every direct graph write (entity create, batch, metadata updates, accepted dream proposals, staged product improvements); metadata is redacted in the form it is serialised in, and a guard keeps only its pattern as written; - hook captures (commit, session summary, pre-compact, handoff), redacted before a title is shortened; - `import` (create, append, overwrite) and the text values of the metadata an import may bring in, including any `replaced_history`; - `replace`: the version it files into `replaced_history` is redacted before the history's size limits are applied; a version too large to keep is left out on its own, the earlier versions stay, and the result says so (`previousVersionDropped`); - the Anthropic memory tool (create, str_replace, insert), on the whole file text; its size limit is checked after redaction. Redaction is idempotent, so editing a file through the memory tool no longer removes lines that follow a redaction marker. `forget` and memory-tool `str_replace` act only on an exact match of the stored text. When only the redacted spelling of the selector is stored, they refuse with one line and change nothing, instead of acting on a line that another secret may also have produced. Text printed into an agent's context — the briefing's ranked sections, index, handoff and task state, the SessionStart injection, pre-edit recall and its guard warnings — and `task_state` over MCP, HTTP and the CLI (reads and write responses), and `/v1/doctor`, are redacted. `recall` and the memory tool's `view` return memories stored before this change as stored. Refs #523, #464, #554
…glued and escaped credentials A memory whose text holds a BEGIN or END line of a private key without the rest no longer takes new metadata text in the same write: the new text cannot be told apart from the rest of the key, so the whole write is refused and nothing is written. memesh pin and memesh unpin add no text, so they are allowed, and they mask every part of a key in that memory's metadata, including history entries that hold one; after that the memory takes new text again. A key or name=value credential written directly after another credential is masked in full, a credential written in JSON escapes inside a duplicate key counts as secret-shaped, and a history entry with a missing or non-text title still masks a key split across its observations.
…art of a private key A write is refused when the memory's stored title, observations, metadata or history hold a BEGIN or END line of a private key without the rest and the write adds any text, including a new metadata key name, whatever its timestamps are. memesh unpin clears such a line in the metadata, also on a memory that was never pinned, and memesh forget --observation removes one from the observations. A task state update is one write again: when its metadata cannot be stored, its new observation is not kept either. Credentials glued into a long chain are masked in time proportional to the text, and text with escapes nested more than eight levels deep is masked on its own without affecting the strings stored with it.
A run of keys glued together, or deeply nested JSON text, no longer makes redaction slow: a search that comes back to where its last match ended skips ahead, and nested levels do not repeat a check already made for the whole text. memesh pin, unpin and task print a refused write as one line (or a JSON error) and exit 1, and the message names how to clear a lone key line in a memory's title as well as in its metadata and observations.
…e-path The history a replaced memory keeps is bounded in one place (replaced-history.ts) for both replace and import overwrite: a version too large for the cap keeps as many observations, then tags, as fit, and its title is cut when even that does not fit, so a replaced version is always kept and the previousVersionDropped result field is gone. Also in this merge: - recall (CLI, MCP, HTTP), the memory tool's view and the dashboard (GET /v1/entities and /v1/entities/:name, history included) mask a credential an older version stored, without changing the stored row; export keeps the stored text; insert numbers lines as view shows them. - forget and str_replace refusals say that shown text is masked and a line is selected by its stored text. - redacting text with long runs of `eyJ` is linear: the JWT search takes each token run's first `eyJ` once instead of rescanning the run from every `eyJ` (300 KB: about 45 s before, milliseconds now; same masking), so the SessionStart hook and the handoff view redact full text within their time budget (#567). - import overwrite redacts the replaced version before bounding the history, so the history stays within its cap. - memesh learn prints one error line and exits 1 when a write is refused, as pin, unpin and task already do. - hooks redact the metadata they store (post-commit file names) and print a guard message with the home directory shown as ~.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…the CRLF fixture The briefing test expected `~/runbook.md`, which is `~\runbook.md` on Windows. The CRLF fixture now replaces the line ending with an anchored pattern.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
token=…assignment, an API key, a private key) is replaced with***REDACTED***on every path that stores memory text:remember(note and structured),learn,import, the Anthropic memory tool,task_state, the hooks' captures, and the graph's own writers. Text inside metadata is redacted too. A private key split across fields or JSON strings is masked as a whole (Credential-shaped text is stored and shown unredacted from remember and commit capture #523, A private key split across JSON strings on separate lines is stored unredacted #565).recall(CLI, MCP, HTTP), the memory tool'sview, the dashboard (GET /v1/entities,GET /v1/entities/:name), the briefing, SessionStart and the pre-edit recall. The stored text is unchanged, andexportreturns it as stored, for backups (Briefing ranked sections do not redact secrets that the index redacts #464, Pre-edit recall prints stored credentials into the agent's context #554).eyJno longer takes tens of seconds. 300 KB took about 45 s and now takes milliseconds, with the same masking (Redaction takes seconds on long text full of eyJ, so the Stop hook can run out of time #567).replaceandimport --merge overwrite. A version too large for the limit is cut to fit instead of dropped.Limits
forgetand the memory tool'sstr_replaceselect a line by its exact stored text, whichexportreturns.Tests
npm run verifyis green on the final tree. New tests cover each write path, the display paths (including the dashboard routes), the SessionStart hook with a 300 KB legacy row and handoff, and the redactor's linear time oneyJruns. Each was checked to fail without its fix.Refs #523, #464, #554, #565, #567