fix(csv): detect the dialect when a quoted field spans several lines - #3985
Merged
cau-git merged 2 commits intoAug 26, 2026
Merged
Conversation
The dialect was sniffed from the first line alone. A quoted field containing a newline is cut mid-quote by that read, so the sniffer sees an unterminated quote, raises, and the backend falls back to a comma. Parsing the file with the wrong dialect under strict=True then aborts the conversion. Retry the sniff on a larger sample when the first line fails, which closes the quote. The first line is still tried first: the sniffer rejects samples whose rows hold different numbers of delimiters, and it infers quotechar from the sample too, so widening it unconditionally changes the dialect detected for ragged files. Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
Contributor
|
✅ DCO Check Passed Thanks @r0h1tb, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
ceberam
requested changes
Aug 13, 2026
Review feedback on docling-project#3985: 4 KiB is plenty for a multi-line quoted field and the sample is now only read when the first line fails to sniff. Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
Contributor
Author
|
Both points addressed in a4f6ba9: sample capped at 4096, and it is now only read when the first-line sniff raises. |
dolfim-ibm
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A CSV whose first field is quoted and contains a newline fails to convert:
convert()sniffs the dialect fromself.content.readline(). That read stops at the newline inside the quoted field, so the sniffer gets'"line one'— an unterminated quote — and raisescsv.Error. The handler falls back tocsv.excel, and reading a semicolon file as comma-delimited understrict=Trueaborts the conversion.Fix
Retry the sniff on a larger sample when the first line fails; the sample closes the quote and the dialect is detected.
The first line is still tried first, deliberately. Sniffing the larger sample unconditionally regresses existing fixtures:
csv.Snifferrejects samples whose rows carry different numbers of delimiters (csv-inconsistent-header,csv-too-many-columns), and it also infersquotecharfrom what it is given — oncsv-too-few-columnsit picks'and strips the quotes from a'b'cell, changing that file's groundtruth. Falling back only on failure leaves every currently-detected dialect untouched.Tests
test_quoted_newline_in_first_fieldintests/test_backend_csv.py. Against unfixed code:tests/test_backend_csv.pygoes 3 passed → 4 passed; no groundtruth was regenerated.ruff checkandruff format --checkare clean.Found while reading the backend, so there is no tracking issue. Related but distinct: #1716 is the single-column sniff failure already handled by the comma fallback.