IO: Fix Windows drive letters misidentified as URI schemes - #3721
Conversation
| scheme='c'. This detects that case so callers can route to the local filesystem. | ||
| Only returns True on Windows — no behavior change on other platforms. | ||
| """ | ||
| return sys.platform == "win32" and len(scheme) == 1 and scheme.isalpha() |
There was a problem hiding this comment.
Windows newb here:
I'm looking at https://stackoverflow.com/questions/446209/possible-values-from-sys-platform. There's a couple other Windows derivatives mentioned. Should we do that for these too?
There was a problem hiding this comment.
Good question! My understanding is that "win32" is sufficient here, the drive letter parsing bug only exists on native Windows where paths look like C:\....
The other sys.platform values from that SO thread don't need coverage because they use POSIX paths, so urlparse never produces a single-letter scheme on them:
- Cygwin (
sys.platform == "cygwin") translatesC:\Users\...to/cygdrive/c/Users/...at the filesystem layer (Cygwin docs) - MSYS2 (
sys.platform == "msys") similarly mapsC:\Usersto/c/Users(MSYS2 docs)
sys.platform == "win32" is also the standard CPython idiom for this, it's how the stdlib distinguishes Windows from POSIX behavior (sys.platform docs), and what pytest recommends for skipif markers.
| scheme='c'. This detects that case so callers can route to the local filesystem. | ||
| Only returns True on Windows — no behavior change on other platforms. | ||
| """ | ||
| return sys.platform == "win32" and len(scheme) == 1 and scheme.isalpha() |
There was a problem hiding this comment.
is it possible to replace
sys.platform == "win32" and len(scheme) == 1 and scheme.isalpha()
with something around os.path.splitdrive?
There was a problem hiding this comment.
Hi @jayceslesar, interesting suggestion, and I think it's reasonable. os.path.splitdrive however operates on full paths, but our function receives the scheme string after urlparse has already extracted it, so we only have "c", not the original path.
We could restructure to check the original path with splitdrive before calling urlparse, but that would change the function signature and all 3 call sites. The current approach keeps the fix minimal and contained within the existing parse flow.
However, refactoring for the splitdrive change is possibly a cleaner design and at a first glance may remove the need for sys.platform and _is_windows_drive_letter. WDYT, should we proceed with this or keep the minimal change?
Edit: I took a closer look and pushed the changes using your suggestion. I think that it's a cleaner approach that avoids some of the uncertainties around sys.platform while achieving the same goal.
On Windows, Python's urlparse treats paths like 'C:\Users\...' as having scheme='c', causing 'Unrecognized filesystem type in URI: c' errors. Uses os.path.splitdrive to detect Windows drive-letter paths before urlparse is called, avoiding the misparse entirely. splitdrive is inherently platform-aware (no-op on Linux/macOS) so no sys.platform check is needed. Fixes all three parse sites (_infer_file_io_from_scheme, PyArrowFileIO.parse_location, FsspecFileIO._get_fs_from_uri) and includes platform-conditional tests. Related: apache#2477, apache#1005
Rationale
On Windows,
urlparse('C:\Users\...')producesscheme='c', causing:This breaks all local file operations for Windows users.
The Fix
A single platform-guarded predicate shared across all three affected parse sites:
When detected, the scheme is remapped to
'file', which routes to PyArrow/fsspec's local filesystem which already handles Windows paths natively.Changes
pyiceberg/io/__init__.py_infer_file_io_from_schemepyiceberg/io/pyarrow.pyparse_locationpyiceberg/io/fsspec.py_get_fs_from_uritests/io/test_io.pytests/io/test_pyarrow.pyparse_locationWindows testWhy all three sites?
They execute at different lifecycle stages and are reachable through independent code paths. Fixing only one (as PR #3161 did) leaves the others broken.
Regression safety
The
sys.platform == 'win32'guard means this is literally invisible on Linux/macOS the predicate short-circuits toFalsewithout inspecting the scheme. Existing CI onubuntu-latestcannot observe any change.Are these changes tested?
Yes. 11 new test cases covering:
@pytest.mark.skipifAll existing tests continue to pass.
Closes
Closes #3720
Related: #2477, #1005