Skip to content

fix: keep non-ascii text intact through the windows clipboard - #74

Merged
backnotprop merged 1 commit into
plannotator:mainfrom
nickgitrabbit:fix/windows-utf8-clipboard
Oct 15, 2026
Merged

backnotprop merged 1 commit into
plannotator:mainfrom
nickgitrabbit:fix/windows-utf8-clipboard

Conversation

@nickgitrabbit

Copy link
Copy Markdown
Contributor

Summary

On Windows, non-ASCII text loses characters on the way through the clipboard, in both directions. This is the Windows path in rust/src/clipboard.rs; macOS and Linux are untouched.

powershell.exe reads and writes a redirected pipe in the console's OEM code page (437 here, 850 on some locales), not UTF-8. The Rust side assumes UTF-8.

Direction Cause Before After
capture (reads the selection from the clipboard) stdout is encoded in code page 437, then decoded as UTF-8 Grão moído is stored as Grao mo�do (ã becomes a, í becomes U+FFFD) Grão moído
copy-context (writes annotations) UTF-8 stdin bytes are decoded as code page 437 clipboard holds U+251C U+00FA where ã should be ã

The typed comment was stored correctly. Only text that crosses the PowerShell pipe was affected, so it shows up as soon as the selection or the comment has an accent. send-archive delivers through Herdr's API, not the clipboard, so its own path was fine, but it forwarded the damaged selection text captured earlier.

Change

  • Read: [Console]::OutputEncoding = New-Object System.Text.UTF8Encoding $false before Get-Clipboard -Raw (UTF-8, no BOM).
  • Write: decode the raw stdin bytes as UTF-8 ourselves, because setting [Console]::InputEncoding does not change how $input is decoded (tried it first, it still produced mojibake). The script keeps what $input | Set-Clipboard did: it drops one trailing line break, joins lines with CRLF, and clears the clipboard on empty input. I measured each of these against the old script.
  • The two script constants are not #[cfg(windows)]: the if cfg!(windows) branches that use them are still compiled on Linux and macOS, so a cfg-gated constant would break those builds (E0425).

Evidence

Live, through Herdr 0.9.5 on Windows 11: selected 3. Açúcar adoça, mas em excesso faz mal. in a pane, ran capture, typed a comment with accents, then copy-context and send-archive.

  • annotations.jsonl holds Açúcar adoça (UTF-16 units 65,231,250,99,97,114), no U+FFFD.
  • The clipboard holds Açúcar and açúcar with CRLF, no U+251C.
  • send-archive delivered the same text to the agent.

Outside Herdr, the exact script strings from this diff were run through powershell.exe 5.1 with CREATE_NO_WINDOW against an independent PowerShell 7 reader and writer: accents, —, ✓ and an emoji (surrogate pair) survive both ways; trailing-newline and empty-input behaviour matches the old script in 4 and 1 cases.

cargo fmt --check and cargo clippy --all-targets -- -D warnings pass. cargo test: 99 pass, 1 fails — store::tests::contention_fails_safely_and_abandoned_locks_recover (store.rs:596), which fails the same way on main without this change.

Tests

Two #[ignore]d tests drive the real clipboard (cargo test clipboard -- --ignored). Each checks one side against a separate PowerShell that moves UTF-16 code units as ASCII numbers, so no code page is involved. A write_clipboard then read_clipboard round trip would not catch this: the two code-page errors cancel out.

Limit, stated plainly: the read test fails without the fix. The write test only fails when powershell.exe starts with a console of its own, which is the case when Herdr runs the plugin with no console. Run from a terminal, the child shares the terminal's console and the write test can pass without the fix.

Merge danger

Two-way door. Windows only; the change is two script strings and tests. Blast radius is the clipboard read in capture and the clipboard write in copy-context / copy-archive and the manager copy on Windows. Not checked: Windows ARM64, PowerShell configured to a UTF-8 system code page (the new scripts do not depend on it).

powershell.exe reads and writes redirected pipes in the console OEM
code page (437 or 850), not UTF-8. A selection such as "Grão moído"
came back as "Grao mo?do" when capturing, and copy-context left
mojibake on the clipboard.

Reading now pins stdout to UTF-8 without a BOM. Writing decodes the raw
stdin bytes as UTF-8, because setting [Console]::InputEncoding does not
change how $input is decoded. The write script keeps what the old
`$input | Set-Clipboard` did: it drops one trailing line break, joins
lines with CRLF, and clears the clipboard on empty input.

The WINDOWS_*_SCRIPT constants are not cfg(windows): the cfg!(windows)
branches that use them are still compiled on Linux and macOS.

Two ignored tests check each direction against a separate PowerShell
that moves UTF-16 code units as ASCII numbers. A write-then-read round
trip cannot catch this, since the two code-page errors cancel out.
@backnotprop
backnotprop merged commit c5e565e into plannotator:main Oct 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants