fix(privacy): redact credentials on every write path and mask them when shown

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes to MeMesh are documented here.
 
 ### Fixed
 
+- Redacting text that holds long runs of `eyJ` without a complete JWT (an error message or log pasted into a memory, for example) no longer takes tens of seconds: 300 KB took about 45 seconds and now takes a few milliseconds, so a hook that redacts such text, or the session start that shows it, stays within its time limit. What is masked is unchanged (#567).
+- Credential-shaped text (a connection-string password, a `token=…` assignment, an API key) is now replaced with `***REDACTED***` on every write path, not only `remember`'s `note` form (names, tags, relations and agent message payloads are identifiers or protocol data and are stored as given): `remember` with structured `title` and `observations`, `learn`, `import` (every merge strategy), the Anthropic memory tool's `create`, `str_replace` and `insert`, `task_state` (its goal, next, blocked and done fields, in the stored record as well as the title), the hooks' capture — including a commit subject the post-commit hook stores as title and first observation, and the pre-compact hook's reason, both redacted before the title is shortened — and the graph's own `createEntity`/`createEntitiesBatch`, so a library caller and the accepted dreamer proposals are covered too (#523). `import` also redacts the text inside the metadata a bundle is allowed to set (for example `verification_scenario`, `success_criteria` and the entries of `replaced_history`), and an accepted product-improvement proposal's `verification_scenario` and `success_criteria` are redacted the same way. A task state stored before this change is redacted when it is shown (in the briefing, at session start and by `memesh task`), so an old goal cannot carry a stored password into the agent's context. Because `***REDACTED***` can be longer than what it replaces, text that passed the `title` and `observations` input caps may be stored longer than those caps; the recall output caps are applied to the stored text, and the memory tool's file-size cap is checked on the redacted text. Also covered: a product-improvement proposal is redacted when it is staged (so `memesh dream show` cannot print a credential), the graph's public writers redact the text inside metadata as well (the `replaced_history` entries a memory already holds are kept as stored), an accepted guard stores its message redacted and a guard proposal is refused — nothing written, still pending — when its examples carry credential-shaped text, `forget` echoes only the redacted form of the text it was sent, the memory tool's `insert` redacts the whole resulting file (so a token placed on the line after an existing `Authorization: Bearer` line is caught), and a private key is masked as a whole region. Text that is, as a whole, one JSON document is redacted by value: every decoded string, key names included, goes through the credential rules, so a key inside a string is masked to the end of that string without touching its quotes. Strings that are stored together are redacted as one set: the strings and key names of one JSON document, a memory's title and observations (`remember`, `import`, the memory tool, hook captures, accepted dream proposals), the fields of one `learn`, task state or product improvement, and the title and observations of one replaced version. When any of them holds only part of a private key (a BEGIN line with no END after it, or an END line with no BEGIN before it, also when written with `\u` escapes inside JSON, as a key split by `readlines()`, across fields or across array elements gives), every string in the set becomes `***REDACTED***`, and in a JSON document every key name too, because which strings hold the rest cannot be told from their order (JSON reorders integer-like keys) (#565). Numbers, booleans, null and the structure stay; key names that land on one name once redacted all stay, each later one as ` (2)`, ` (3)`…, so no value is dropped. Metadata is redacted string by string, so fields such as `trust`, `kind` and timestamps are never masked; new metadata that would still hold part of a private key, or a key name that holds a credential, is refused and nothing is written (text a memory already holds is not checked again). Part of a private key next to new text in one write is refused as a whole too, and nothing is written: a BEGIN or END line in the title or observations with text or key names in the metadata, or such a line that a memory stored before this change already holds in its title, observations, metadata or `replaced_history`, beside any new text (a title, an observation, a metadata value or key name). The new text cannot be told apart from the rest of a key; a lone BEGIN or END line is not proof of a key, and the message says so. `memesh unpin --name <name>` adds no text, so it is allowed on such a memory (pinned or not), and it masks every part of a key in its metadata, including the history entries that hold one (each redacted as its own set, its timestamp and other fields kept; every other history entry stays as stored); a line among the observations is removed with `memesh forget --name <name> --observation "<that line>"`, and such a title is replaced with `memesh remember --name <name> --type <its type> --title "<new title>"`. After that the memory takes new text again. A `learn` whose error text is nothing but credentials once redacted is refused too, since it could not be told apart from another such lesson. A document with no escape (`\`) in it and nothing to redact comes back byte for byte; any other is written again compactly with `JSON.stringify`, which keeps every number as spelled and, of duplicate keys, only the last (`JSON.parse` drops the others, so an earlier duplicate is never stored). A dream proposal or guard example counts as credential-shaped only when something in it is masked (a credential in a duplicate JSON key included, also when it is written in `\u` escapes; text whose escapes nest more than eight levels deep cannot be read and is stored as `***REDACTED***` on its own), not when its JSON is merely written again. A JSON document nested too deep to read (about a thousand levels) cannot be checked and is stored as `***REDACTED***` as a whole. In other text, a private-key region is sensitive as a whole: from the `-----BEGIN … PRIVATE KEY-----` header through the next `-----END … PRIVATE KEY-----` line, whatever is inside (a hard-wrapped key, line prefixes, blank lines, junk characters, any kind of line break). A header with no END line after it has no trustworthy end, so everything from it to the end of the text is masked: prose that merely quotes a header loses the text after it, and editing a note stored before this change that does so masks it from the header on. Text before the header and text after a genuine END stay, and text with no header is untouched. A session handoff stored before this change is redacted when it is shown, like the task state, and so is a task state read back through the `task_state` tool, `GET /v1/task-state` or `memesh task --json`, and the state a `task_state` write or `memesh task … --json` answers with. A memory written before this change that still holds a credential stays removable and editable: `forget` and the memory tool's `str_replace` find the line by its text exactly as stored and never echo it back. `recall`, the memory tool's `view` and the dashboard (`GET /v1/entities` and `GET /v1/entities/:name`, its `replaced_history` included) show such a memory masked (the stored text is unchanged, and `export` returns it as stored, for backups), the memory tool's `insert` numbers lines as `view` shows them, an `import --merge overwrite` redacts the version it files into `replaced_history` before applying the history's size limit, the capture hooks redact the text in the metadata they store (a commit's file names, for example), and the guard warnings the hooks inject have home-directory paths replaced with `~` like every other memory line. `memesh learn` prints a refused write as one line (or a JSON error) and exits 1. The doctor report redacts each of its text values before it is serialised, so a key in one of them is masked to the end of that value and a key with `\r\n` or tab-indented lines does not reach an issue body either. Redaction repeats until the text stops changing (credentials glued together with no separator, such as an AWS key id followed directly by `password=…` or a GitHub token followed directly by an `sk-` key, are caught). Redacting text that is already redacted therefore changes nothing, and a memory-tool edit never removes the lines after an earlier `***REDACTED***`; it also means a private key written into a memory-tool file one line per call is recognised as a key only once its END line is in the file. An accepted guard's pattern (a string) is never rewritten by a later metadata write (a pattern that detects credentials looks like one), while its message, its examples and any other text in it are redacted like other metadata; a guard message stored before this change is redacted when a hook prints it; a guard whose stored pattern no longer compiles is recorded as an error outcome by both the Bash and the Edit/Write guard hooks instead of being skipped silently; and a guard proposal is refused only when an example carries credential-shaped text — the message names `memesh dream reject`, since nothing edits a proposal. Metadata is redacted as it will be stored, so a value with its own `toJSON()` or a dictionary without a prototype cannot slip a credential past it; a `replaced_history` handed in by a caller — including one an updater pushes onto the stored list, and one that is not a list — is redacted like any other text, while the entries a memory already holds are kept exactly as they are; the version `replace` files away is new history and is redacted too, before the history's count and byte limits are applied; a version that still exceeds the byte limit with its observations and tags cut has its title cut until it fits, dropping a half emoji the cut leaves behind, so a replaced version is always kept. `forget` and the memory tool's `str_replace` act only on an exact match of the stored text. Many different texts redact to the same `***REDACTED***`, so when only the redacted spelling of what they were sent is stored they refuse the call as ambiguous and change nothing ("No exact stored-text match …": `recall` and `view` show credential-shaped text masked, so a line is selected by its stored text, which `export` returns), instead of removing or editing an unrelated redacted line and reporting success; a line stored redacted is removed or edited by its stored text. A selector that matches neither spelling gets the same not-found answer as before. A staged product-improvement whose title or success criterion grows past its input cap through redaction is no longer refused. A credential that spans lines (a private key block, a `Bearer` token on the next line) is redacted as a whole before the memory tool splits the text into lines, and the tool's `str_replace` reply shows the stored text. Names and tags are never rewritten; metadata values are, and so is a metadata key that holds a credential. An imported `replaced_history` that no longer fits its size limits once its text is redacted is left out, and the import says so in its `errors`. Memories stored before this change keep the text they were stored with (shown masked, as above) with one exception: a memory-tool `str_replace` or `insert` rewrites the whole file, so an old credential on an untouched line of that file is redacted too — and so is ordinary text on an untouched line that happens to look like a credential (for example a word starting `sk-` followed by several more characters, such as `sk-learn-style`). Two credentials written with nothing between them (`ghp_…ghp_…`, two JWTs, `Bearer` tokens) are both masked, at any length: the search looks again inside each match (skipping ahead once it finds the same match again, except for JWTs and SendGrid keys, which are made of segments), so the first one's match no longer swallows the start of the second and leaves the rest of it in the text. On text where overlapping matches would cost more than four passes over it, everything from that point on is masked rather than kept.
+- The pre-edit recall hook (the memories shown before an Edit or Write) redacts a credential in a memory's name and first observation before printing them, and before shortening the line, so an old memory cannot carry a stored password into the agent's context (#554).
+- Every memory line in the injected briefing — the ranked sections such as "Decisions and direction" and the lessons, not only the durable-memory index — has secrets and home-directory paths redacted, from both the MCP/CLI `briefing` and the SessionStart hook. One memory could appear twice in one block, redacted under the index heading and unredacted a few lines above it (#464). The whole first observation is redacted before it is shortened for the line, so a credential longer than the shortening window no longer leaks its prefix.
 - Memories about a file are shown before an edit in every session again. Once any session (or another agent on the same machine) had been shown them, every other session editing that file got nothing, and the hook record said "nothing to recall". Each session now keeps its own list (a subagent keeps one of its own), a repeat is recorded as throttled, and a list that cannot be written or cleaned up is recorded as an error instead of failing silently (#521).
 - Importing with `overwrite` (MCP `import`, `POST /v1/import`, `memesh import --merge overwrite`) now keeps the memory's previous observations, tags and title in its `replaced_history`, the same way `remember` with `replace: true` does. Before, the old content was erased with no way back. Importing a file that names one memory more than once is refused before anything is written (#530).
 - A message sent through the MCP `message` tool keeps every `null` in its JSON payload. Before, the MCP boundary removed each null-valued key at any depth of the payload while `send` still reported success, so the recipient got different data than the CLI or HTTP would have stored (#517). A null-valued top-level tool parameter still means "left blank", except `payload`, whose null is a value (#553).