Skip to content

Only write ImageSeries.num_samples when it was explicitly provided - #2239

Merged
rly merged 6 commits into
NeurodataWithoutBorders:devfrom
adityasingh2400:fix-2227-imageseries-num-samples
Aug 31, 2026
Merged

Only write ImageSeries.num_samples when it was explicitly provided#2239
rly merged 6 commits into
NeurodataWithoutBorders:devfrom
adityasingh2400:fix-2227-imageseries-num-samples

Conversation

@adityasingh2400

Copy link
Copy Markdown
Contributor

Motivation

Fix #2227.

ImageSeries.num_samples is a property that falls back to the inherited TimeSeries.num_samples, i.e. len(data), when the user did not supply a value. There is no custom mapping for that dataset, so the default ObjectMapper read the property and persisted the derived value. Every ImageSeries (and OpticalSeries, OnePhotonSeries, TwoPhotonSeries) built from internal data therefore wrote a num_samples dataset the user never set, and because the derived value is a Python int and the schema dataset is uint32, HDMF widened it to uint64 and emitted DtypeConversionWarning: Spec 'ImageSeries/num_samples': Value with data type int64 is being converted to data type uint64 (min specification: uint32). That warning fires across the image and ophys integration tests today.

The schema documents num_samples as the frame count that cannot otherwise be recovered, which is the format='external' plus starting_time/rate case, so writing a value derived from len(data) is redundant. This matches the conclusion in #2215 that persisting it for internal data was an oversight.

Fix

ImageSeriesMap now maps the write path of the num_samples spec to the private _num_samples, which holds a value only when num_samples was explicitly passed to the constructor. When it was not set, nothing is written. The read path is untouched, so a file that does contain num_samples still maps it back to the num_samples constructor argument and round-trips unchanged. The in-memory ImageSeries.num_samples property is also unchanged and still returns the frame count derived from the data.

An object_attr override additionally casts an explicitly provided, in-range value to uint32 so it is written with the schema dtype rather than being widened to uint64. Out-of-range values are passed through untouched so HDMF still reports the mismatch.

How to test the behavior?

import numpy as np, warnings, h5py
from datetime import datetime
from dateutil.tz import tzlocal
from pynwb import NWBFile, NWBHDF5IO
from pynwb.image import ImageSeries

nwbfile = NWBFile("s", "i", datetime.now(tzlocal()))
nwbfile.add_acquisition(ImageSeries(name="img", data=np.zeros((10, 5, 5), dtype=np.uint8), unit="n.a.", rate=1.0))
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    with NWBHDF5IO("test.nwb", "w") as io:
        io.write(nwbfile)
print([str(x.message) for x in w])   # before: one DtypeConversionWarning, after: []
with h5py.File("test.nwb") as f:
    print(list(f["acquisition/img"].keys()))   # before: [..., 'num_samples', ...], after: no num_samples

Four new tests in tests/integration/hdf5/test_image.py cover this. They fail on dev and pass with the change:

FAILED tests/integration/hdf5/test_image.py::TestImageSeriesWithNumSamplesIO::test_num_samples_written_as_uint32
FAILED tests/integration/hdf5/test_image.py::TestImageSeriesDerivedNumSamplesIO::test_derived_num_samples_not_written
FAILED tests/integration/hdf5/test_image.py::TestImageSeriesNumSamplesWriteWarnings::test_no_warning_for_derived_num_samples
FAILED tests/integration/hdf5/test_image.py::TestImageSeriesNumSamplesWriteWarnings::test_no_warning_for_explicit_num_samples
4 failed, 4 passed, 9 deselected

After the change, pytest tests/integration/hdf5/test_image.py tests/integration/hdf5/test_ophys.py passes with zero num_samples dtype-conversion warnings, down from the previous count.

I deliberately left out the consistency check you suggested in #2215, where an explicitly provided num_samples on an internal ImageSeries should be validated against the number of frames in the data. Happy to add it here or in a follow-up if you would like it.

Checklist

  • Did you update CHANGELOG.md with your changes?
  • Have you checked our Contributing document?
  • Have you ensured the PR clearly describes the problem and the solution?
  • Is your contribution compliant with our coding style? This can be checked running ruff check . && codespell from the source directory.
  • Have you checked to ensure that there aren't other open Pull Requests for the same change?
  • Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

ImageSeries.num_samples falls back to the inherited TimeSeries.num_samples
property, len(data), when the user did not supply a value. The default
ObjectMapper persisted that derived value, so every ImageSeries with internal
data wrote a num_samples dataset the user never set, and HDMF widened the
Python int to uint64 against the schema's uint32 dataset and emitted a
DtypeConversionWarning.

Map the write path to the private _num_samples so the dataset is written only
when num_samples was explicitly provided, and cast an in-range explicit value to
uint32 so it is written with the schema dtype. The read path is unchanged, so
files that do contain num_samples still map it back to the constructor argument.

Fix NeurodataWithoutBorders#2227
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.17%. Comparing base (f378275) to head (47c0a1f).

Files with missing lines Patch % Lines
src/pynwb/io/image.py 83.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #2239      +/-   ##
==========================================
- Coverage   96.22%   96.17%   -0.06%     
==========================================
  Files          30       30              
  Lines        2991     3003      +12     
  Branches      435      437       +2     
==========================================
+ Hits         2878     2888      +10     
- Misses         64       65       +1     
- Partials       49       50       +1     
Flag Coverage Δ
integration 75.02% <83.33%> (+0.03%) ⬆️
unit 86.41% <33.33%> (-0.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rly
rly self-requested a review August 31, 2026 22:39
Filtering the collected warnings on the message text ties the assertions
to HDMF's wording, so a reworded warning would make them pass trivially.

Also shortens the CHANGELOG entry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rly

rly commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Thanks @adityasingh2400 . I made a couple minor changes. This is good to go.

@rly
rly enabled auto-merge (squash) August 31, 2026 22:52
@rly
rly merged commit b5c3b81 into NeurodataWithoutBorders:dev Aug 31, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ImageSeries writes auto-derived num_samples, causing a DtypeConversionWarning

2 participants