Repository navigation
fix: keep non-ascii text intact through the windows clipboard - #74
Merged
backnotprop merged 1 commit intoOct 15, 2026
Merged
Conversation
powershell.exe reads and writes redirected pipes in the console OEM code page (437 or 850), not UTF-8. A selection such as "Grão moído" came back as "Grao mo?do" when capturing, and copy-context left mojibake on the clipboard. Reading now pins stdout to UTF-8 without a BOM. Writing decodes the raw stdin bytes as UTF-8, because setting [Console]::InputEncoding does not change how $input is decoded. The write script keeps what the old `$input | Set-Clipboard` did: it drops one trailing line break, joins lines with CRLF, and clears the clipboard on empty input. The WINDOWS_*_SCRIPT constants are not cfg(windows): the cfg!(windows) branches that use them are still compiled on Linux and macOS. Two ignored tests check each direction against a separate PowerShell that moves UTF-16 code units as ASCII numbers. A write-then-read round trip cannot catch this, since the two code-page errors cancel out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Windows, non-ASCII text loses characters on the way through the clipboard, in both directions. This is the Windows path in
rust/src/clipboard.rs; macOS and Linux are untouched.powershell.exereads and writes a redirected pipe in the console's OEM code page (437 here, 850 on some locales), not UTF-8. The Rust side assumes UTF-8.capture(reads the selection from the clipboard)Grão moídois stored asGrao mo�do(ãbecomesa,íbecomes U+FFFD)Grão moídocopy-context(writes annotations)ãshould beãThe typed comment was stored correctly. Only text that crosses the PowerShell pipe was affected, so it shows up as soon as the selection or the comment has an accent.
send-archivedelivers through Herdr's API, not the clipboard, so its own path was fine, but it forwarded the damaged selection text captured earlier.Change
[Console]::OutputEncoding = New-Object System.Text.UTF8Encoding $falsebeforeGet-Clipboard -Raw(UTF-8, no BOM).[Console]::InputEncodingdoes not change how$inputis decoded (tried it first, it still produced mojibake). The script keeps what$input | Set-Clipboarddid: it drops one trailing line break, joins lines with CRLF, and clears the clipboard on empty input. I measured each of these against the old script.#[cfg(windows)]: theif cfg!(windows)branches that use them are still compiled on Linux and macOS, so acfg-gated constant would break those builds (E0425).Evidence
Live, through Herdr 0.9.5 on Windows 11: selected
3. Açúcar adoça, mas em excesso faz mal.in a pane, rancapture, typed a comment with accents, thencopy-contextandsend-archive.annotations.jsonlholdsAçúcar adoça(UTF-16 units 65,231,250,99,97,114), no U+FFFD.Açúcarandaçúcarwith CRLF, no U+251C.send-archivedelivered the same text to the agent.Outside Herdr, the exact script strings from this diff were run through
powershell.exe5.1 withCREATE_NO_WINDOWagainst an independent PowerShell 7 reader and writer: accents,—,✓and an emoji (surrogate pair) survive both ways; trailing-newline and empty-input behaviour matches the old script in 4 and 1 cases.cargo fmt --checkandcargo clippy --all-targets -- -D warningspass.cargo test: 99 pass, 1 fails —store::tests::contention_fails_safely_and_abandoned_locks_recover(store.rs:596), which fails the same way onmainwithout this change.Tests
Two
#[ignore]d tests drive the real clipboard (cargo test clipboard -- --ignored). Each checks one side against a separate PowerShell that moves UTF-16 code units as ASCII numbers, so no code page is involved. Awrite_clipboardthenread_clipboardround trip would not catch this: the two code-page errors cancel out.Limit, stated plainly: the read test fails without the fix. The write test only fails when
powershell.exestarts with a console of its own, which is the case when Herdr runs the plugin with no console. Run from a terminal, the child shares the terminal's console and the write test can pass without the fix.Merge danger
Two-way door. Windows only; the change is two script strings and tests. Blast radius is the clipboard read in
captureand the clipboard write incopy-context/copy-archiveand the manager copy on Windows. Not checked: Windows ARM64, PowerShell configured to a UTF-8 system code page (the new scripts do not depend on it).