Skip to content

fix: stop treating an embedded NUL as end of input - #6

Merged
mazzma12 merged 1 commit into
mainfrom
fix/nul-byte-truncation
Sep 3, 2026
Merged

fix: stop treating an embedded NUL as end of input#6
mazzma12 merged 1 commit into
mainfrom
fix/nul-byte-truncation

Conversation

@mazzma12

@mazzma12 mazzma12 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Fixes #5

nextBy/lookAhead returned rune 0 past the end and isEOF tested ch == 0, so a literal NUL in the input was indistinguishable from end-of-input and every !isEOF(ch) scan loop stopped at the first one, dropping the rest with no token and no error.

Took direction 1 (sentinel rune(-1), which decoding can never produce): it keeps the existing "the rune the loop already carries decides" shape for a 3-line change, where deriving EOF from the cursor also means reordering Scan's switch, since peek still has to return something no rune predicate matches or scanUnknown spins at cursor == len. A NUL now falls through to UNKNOWN and scanning continues, matching \x01/\x1f/\x7f.

Tests: NUL mid-statement, first, last, alone, repeated, and inside each of the four scan loops (string literal, quoted identifier, line comment, block comment), plus an equivalence test that a NUL lexes exactly like the other control bytes. Existing suite passes unchanged; I also diffed this build against main over 139k real NUL-free queries (all three sqlprocessor modes) and 200k generated hostile ones across five dialects — byte-identical.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mazzma12
mazzma12 merged commit a69a735 into main Sep 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedded NUL byte silently truncates the token stream (isEOF conflates 0x00 with end-of-input)

2 participants