Repository navigation
fix: queue heredoc bodies per command line and drain in opener order - #18
Merged
Merged
Conversation
These fail today with 'Expected a command word': the parser only lets a redirect consume a body token directly after the opener, so any separator or second heredoc on the same line leaves body tokens orphaned after the command. Reference shapes adapted from mvdan/sh filetests.
The tokenizer emits one heredoc-body token per queued heredoc right after the newline ending a command line, but parseRedirect only let a redirect consume a body token directly following its opener. With two heredocs on one command (cat << A << B) the first body token sat between the second redirect and its target, and with a separator (;, &&, ||, |) after the opener nothing ever consumed the body token; both ended with an orphan token and an 'Expected a command word' parse error. parseRedirect now queues heredoc redirects instead of looking for the body inline, and a drain step in skipSeparators/skipCaseSeparators attaches body tokens to their redirects in opener order as they surface. This mirrors mvdan/sh's per-line heredoc queue. Orphan body tokens with no pending redirect are left in the stream for the caller to report, as before. A redirect-only statement now parses with its heredoc body attached instead of throwing.
aliou
marked this pull request as ready for review
September 4, 2026 07:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Automated pull request by Pi, model
aperture/synthetic/hf:zai-org/GLM-5.3-Flash. Implemented and validated empirically againstaliou/shon branchfix/heredoc-statement-boundary. See the local run log for the full investigation.Summary
Second layer of the heredoc statement-boundary work, stacked on #17. Two failure groups remained after that fix — both empirically confirmed throwing
Expected a command wordon its tip:cat << A << Bwith bodiesfoo/A/bar/Bthrows; the parser only lets a redirect consume a heredoc-body token that directly follows its opener, so the first body token sits between the second redirect and its target, and bodies bind to the wrong redirects.cat << EOF; echo hi,cat << EOF && echo done,cat << EOF | grep foo: nothing on the right of the opener consumes the body token queued after the newline, so the body surfaces where a command is expected and throws.Both are valid bash and are covered upstream by mvdan/sh's
syntax/filetests_test.go(e.g.<<EOF1 <<EOF2 | c && d,foo <<EOF && { ... },f1 <<EOF1; f2 <<EOF2). Several failing tests here are adapted from those cases.Changes
src/parser/heredoc.test.ts— commit 1/2: nine failing tests covering the cases above plus in-compound variants. Do not modify these to make the fix pass.src/parser/parser.ts— commit 2/2: the fix.parseRedirectqueues heredoc redirects inpendingHeredocsinstead of looking for the body inline; adrainPendingHeredocsstep inskipSeparators/skipCaseSeparatorsattaches body tokens to their redirects in opener order as they surface (mirroring mvdan/sh's per-line heredoc queue). Orphan body tokens with no pending redirect are left in the stream and still error, as before..changeset/legal-owls-check.md— patch changeset.Behavior notes:
<<EOF1 <<EOF2now parse with bodies attached instead of throwing (same shape as mvdan/sh's redirect-only statements).&&/|open heredocs (a <<A && b <<B:agetsx\n,bgetsy\n).Verification
pnpm typecheck✅pnpm lint✅pnpm test✅ (290 passed; the 9 new tests fail on fix: stop heredoc commands from swallowing the following statement #17's tip and pass here; all 281 prior tests untouched)&&/|, multi-heredoc on a pipeline's left, case items, command substitution, EOF-terminated delimiters, empty bodies, comments after delimiters — all parse with correct body assignment.Known limit (pre-existing, unchanged): redirects after compound commands (
while ...; done <<IN) still don't attach to the compound — that needs compound-command redirect support first.Stacked on #17
Run host:
sh-d9e31b· Session:01a062c1-c9e1-71cb-b132-aafe96837313· Model:aperture/synthetic/hf:zai-org/GLM-5.3-Flash(validation),aperture/neuralwatt/kimi-k3(implementation)