fix: preserve whitespace-only cells in the XLS reader#177
Merged
Conversation
DataFrame.LoadCsv parses via Microsoft.VisualBasic TextFieldParser, whose TrimWhiteSpace is true and cannot be configured through LoadCsv, so it strips whitespace from every field (even quoted). As a result a cell containing only spaces was read back as an empty string. The XLS write path (CsvUtils.EscapeCsvValues) now wraps a whitespace-only value in a Unicode private-use guard character so the field's edges are non-whitespace and survive parsing. CsvVirtualDataTable strips the guard afterwards, and only when the guarded content is itself whitespace-only, so real values are never affected. Values without the guard (e.g. every CSV-provider value) are returned unchanged. Adds the shared Reader_ShouldPreserveWhitespaceOnlyCell test and a withSpaceCell.xlsx fixture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Move the guard sentinel to a shared Data.Common CsvWhitespaceGuard class (was a public const on the CsvVirtualDataTable paging read class). - Gate guard stripping behind stripWhitespaceGuard, set only by the XLS reader, so other providers never strip a value that merely looks guard-shaped. This makes the "never corrupt data I did not encode" property structural rather than probabilistic. - Parameterize the shared whitespace test by table name; add a focused CSV test proving a guard-shaped value is returned unchanged when stripping is disabled. - Add a mixed numeric+whitespace fixture/test documenting that preserving a whitespace cell makes an otherwise-numeric column read as text (XLS read path only). - Document the whitespace-only scope limitation in CsvUtils. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SummarySummary
CoverageData.Common - 77.4%
Data.Csv - 90.9%
Data.Json - 81.5%
Data.Tests.Common - 88.6%
Data.Xls - 66.5%
Data.Xml - 86.8%
EFCore.Common - 68.5%
EFCore.Csv - 68.2%
EFCore.Json - 71.8%
EFCore.Xml - 73%
|
DaveRMaltby
approved these changes
Jul 3, 2026
DaveRMaltby
left a comment
DaveRMaltby
left a comment
Contributor
There was a problem hiding this comment.
Dominic walked me through all the changes and they look good. It is unfortunate that we have to use marked in the cell values that only have whitespace, but we cannot avoid the behavior of the DataFrame.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A cell containing only whitespace (e.g. two spaces) was read back as an empty string by the XLS provider.
Root cause:
DataFrame.LoadCsvparses CSV viaMicrosoft.VisualBasic.FileIO.TextFieldParser, whoseTrimWhiteSpaceproperty istrue, is not configurable throughLoadCsv, and trims leading/trailing whitespace from every field — including quoted fields. So a whitespace-only cell reached the reader as"".Fix: The XLS write path (
CsvUtils.EscapeCsvValues) wraps a whitespace-only value in a Unicode private-use guard character (U+E000) so the field's edges are non-whitespace and surviveTextFieldParser.CsvVirtualDataTablestrips the guard after parsing — and only when the guarded content is itself whitespace-only, so a real value that merely began/ended with the guard character is never corrupted. Any value without the guard (e.g. every CSV-provider value) is returned unchanged, so the CSV provider is unaffected.Tests
DataReaderTests.Reader_ShouldPreserveWhitespaceOnlyCell<TFileParameter>and awithSpaceCell.xlsxfixture, wired throughXlsDataReaderTests(mirrors the existing delegation pattern).Data.Xls.Tests: 35 passed, 5 skipped, 0 failed.Data.Csv.Tests: 129 passed, 0 failed (guard decode is a no-op for CSV values).🤖 Generated with Claude Code