In one 100-variable batch run, ocean.tauuo crashed while reading a source
file. The other 99 passed.
Key: ('open_dataset-tau_x-c94b7655...', 0, 0, 0)
File "access_moppy/base.py", line 1927, in write_next
destination[slices] = future.result()
File "src/netCDF4/_netCDF4.pyx", line 1750, in _get_format
RuntimeError: NetCDF: Not a valid ID
file: .../PI-CNP-concentrations-full/output005/ocean/ocean-2d-tau_x-1mon-mean-y_0106.nc
run: /scratch/tm70/yz9299/piControl_parallel_thorough_cc_check_2 (job 177365482)
The error is raised in nc_inq_format, i.e. after nc_open returned, so
the ncid was already invalid — a corrupted netCDF-C file-table entry, not a bad
file.
Not the data
- The file reads fine standalone (valid HDF5,
NETCDF4_CLASSIC, values OK).
- ACL is byte-identical to the neighbouring
tau_y file.
ocean.tauvo — same pipeline, same directory, adjacent file — passed in the
same batch.- Re-running the identical job passes (job
177370214, exit 0).
Suspected cause
A race on concurrent netCDF-C calls inside one worker process.
_parallel_open_is_safe() already names this exact symptom, but its
one-thread-per-worker guard only covers the compute thread: nc_close also
runs from GC on the event-loop thread when _write_dask_slices releases
futures. Consistent with the timing — the failing block is block 0 of the 6th
file, after five files had been read and released.
Two lock defects that would allow it, both demonstrated:
SerializableLock._locks is a WeakValueDictionary. On a worker the token
comes from the client process and nothing holds it, so once the last strong
reference dies the next deserialisation builds a new threading.Lock
under the same token and mutual exclusion silently stops.CombinedLock.acquire(blocking=False) is all(acquire(l, False) for l in self.locks). On short-circuit, lock 0 stays held;
CachingFileManager.__del__ sees False and skips its release(), so it
leaks permanently. Upstream xarray bug (2026.1.0); shows up as a hang, not
this crash, but it is the same code path.
In one 100-variable batch run,
ocean.tauuocrashed while reading a sourcefile. The other 99 passed.
The error is raised in
nc_inq_format, i.e. afternc_openreturned, sothe ncid was already invalid — a corrupted netCDF-C file-table entry, not a bad
file.
Not the data
NETCDF4_CLASSIC, values OK).tau_yfile.ocean.tauvo— same pipeline, same directory, adjacent file — passed in thesame batch.
177370214, exit 0).Suspected cause
A race on concurrent netCDF-C calls inside one worker process.
_parallel_open_is_safe()already names this exact symptom, but itsone-thread-per-worker guard only covers the compute thread:
nc_closealsoruns from GC on the event-loop thread when
_write_dask_slicesreleasesfutures. Consistent with the timing — the failing block is block 0 of the 6th
file, after five files had been read and released.
Two lock defects that would allow it, both demonstrated:
SerializableLock._locksis aWeakValueDictionary. On a worker the tokencomes from the client process and nothing holds it, so once the last strong
reference dies the next deserialisation builds a new
threading.Lockunder the same token and mutual exclusion silently stops.
CombinedLock.acquire(blocking=False)isall(acquire(l, False) for l in self.locks). On short-circuit, lock 0 stays held;CachingFileManager.__del__seesFalseand skips itsrelease(), so itleaks permanently. Upstream xarray bug (2026.1.0); shows up as a hang, not
this crash, but it is the same code path.