Skip to content

Prepare for release of PyNWB 4.2.0 - #2246

Open
rly wants to merge 13 commits into
devfrom
rel/4.2.0
Open

Prepare for release of PyNWB 4.2.0#2246
rly wants to merge 13 commits into
devfrom
rel/4.2.0

Conversation

@rly

@rly rly commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Prepare for release of PyNWB 4.2.0.

Also fixes several things found while working through the checklist, and expands the checklist itself:

  • Fixed the tag filter on the "Run all tests" workflow
  • Fixed some -upgraded entries in the "Run all tests" matrices
  • Cleaned up and fixed errors in make_a_release.rst
  • Fixed and shortened the runtime of the resources_streaming.py tutorial

Version number

The semantic-versioning checklist item flags "removes or narrows accepted input" as a breaking change. #2235 narrows TimeSeries.control_description and the other array-valued fields from Iterable to array types, so a str, set, range, or generator is no longer accepted, and it changes TimeSeries.get_timestamps from raising TypeError to raising ValueError.

4.2.0 stays a minor release because those iterables were accepted by the type check but did not produce a correct file, and get_timestamps already raised for the same input.

Before merging:

  • Make sure all PRs to be included in this release have been merged to dev.
  • Major and minor releases: Update the dependency version bounds in pyproject.toml and the pinned versions in environment-ros3.yml to the latest as needed. The zarr and numcodecs upper bounds in the zarr extra track hdmf-zarr's own bounds. Check hdmf-zarr's requirements before touching them rather than raising them to the latest release. (dandi is pinned to 0.78.0 and installed from conda-forge rather than PyPI, since conda-forge publishes cp314 builds for every platform the ROS3 jobs use. The defaults channel was dropped from environment-ros3.yml because every package in the environment resolves from conda-forge. Both were checked with conda env create --dry-run for linux-64, osx-arm64, and win-64.)
  • Major releases: Remove the deprecated functionality slated for removal in this version. (n/a, minor release; nothing is marked for removal before PyNWB 5.0)
  • Check legal file dates and information in Legal.txt, license.txt, README.rst, docs/source/conf.py, and any other locations as needed
  • Update pyproject.toml as needed
  • Confirm the version number matches the severity of the changes, following semantic versioning. A changelog entry that removes or narrows accepted input, changes a return type, or turns a warning into an exception is a breaking change and belongs in a major release. Two cases are exempt from that rule and may go in a minor release: input that was accepted but could not be written or read back correctly, and a change in the type of an exception that was already raised for the same input. Record the reasoning in this pull request. (See "Version number" above.)
  • Update README.rst as needed
  • Update src/pynwb/nwb-schema submodule as needed. Check the version number and commit SHA manually. Make sure we are using the latest release and not the latest commit on the main branch.
  • Update changelog (set release date) in CHANGELOG.md and any other docs as needed
  • Cross-check the changelog against git log --oneline <previous-tag>..HEAD so that every merged PR is represented.
  • Confirm the release notes will be extracted from the new heading by running the awk from deploy_release.yml with hdr="## PyNWB <version>" against CHANGELOG.md and checking that the output is non-empty and stops before the previous version's heading.
  • Run tests locally including gallery and validation tests, and inspect all warnings and outputs (python test.py -v -p -i -b -w -x > out.txt 2>&1). Try to remove all warnings.
  • Test docs locally and inspect all warnings and outputs cd docs; make clean && make html
  • Build and test-install the distributions from a clean clone (git clone --recurse-submodules into a temp directory): run python -m build, then install the .whl and the .tar.gz into separate fresh virtual environments and confirm import pynwb, that nwb-schema/core ships with the expected schema version, and that writing and validating a minimal NWBFile succeeds.
  • After pushing this branch to GitHub, manually trigger the "Run all tests" GitHub Actions workflow on this branch by going to https://github.com/NeurodataWithoutBorders/pynwb/actions/workflows/run_all_tests.yml, selecting "Run workflow" on the right, selecting this branch, and clicking "Run workflow". Make sure all tests pass, and check the ROS3 streaming jobs specifically, since those need a conda environment and are not run locally. https://github.com/NeurodataWithoutBorders/pynwb/actions/runs/33644415245
  • Check that the readthedocs build for this PR succeeds (see the PR check)

After merging:

  1. Create release by following steps in docs/source/make_a_release.rst or use alias git pypi-release [tag] if set up
  2. After the CI bot creates the new release (wait ~10 min), check the release notes on the GitHub releases page. The workflow fills them from this version's section of CHANGELOG.md, and falls back to auto-generated notes if that section is empty or its heading does not match the tag.
  3. Check that the readthedocs "stable" build runs and succeeds
  4. Either monitor conda-forge/pynwb-feedstock for the regro-cf-autotick-bot bot to create a PR updating the version of PyNWB to the latest PyPI release, usually within 24 hours of release, or manually create a PR updating recipe/meta.yaml with the latest version number and SHA256 retrieved from PyPI > PyNWB > Download Files > View hashes for the .tar.gz file. Re-render and update dependencies as needed.

🤖 Generated with Claude Code

Set the 4.2.0 release date in CHANGELOG.md.

Update the pinned versions in environment-ros3.yml to the latest releases
available on conda-forge and PyPI.

Ignore the zarr tutorial output directory and the streaming tutorial's
fsspec cache directory, which the gallery writes into the working tree.

Fix the tag filter on the "Run all tests" workflow. GitHub Actions matches
tags with glob patterns, so the regular expression it held matched no tag.
Use the same MAJOR.MINOR.PATCH glob as the "Deploy release" workflow.

Drop the "-upgraded" entries from the "Run all tests" matrices. In that
workflow an unsuffixed name denotes the upgraded tox environment, so each
of these ran the same tox environment and operating system as its
unsuffixed neighbor.

Expand the release pull request template. New items cover confirming the
version number against semantic versioning, cross-checking the changelog
against the commits since the previous tag, confirming the release notes
extract from the new heading, building and test-installing the
distributions from a clean clone, and triggering the "Run DANDI read
tests" workflow. The local test command gains -x for the pynwb.validate
tests and drops -r, with the ROS3 jobs checked in CI instead. The
dependency bounds item records that the zarr extra tracks hdmf-zarr's
bounds, and the regro-cf-autotick-bot note now names PyNWB.

Replace the checklist summary in make_a_release.rst with a pointer to the
template, so the checklist has one authoritative copy, and name
pyproject.toml as the source of the conda-forge requirements.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.14%. Comparing base (b5c3b81) to head (ff99f7d).

❗ There is a different number of reports uploaded between BASE (b5c3b81) and HEAD (ff99f7d). Click for more details.

HEAD has 5 uploads less than BASE
Flag BASE (b5c3b81) HEAD (ff99f7d)
unit 4 3
integration 4 0
Additional details and impacted files
@@             Coverage Diff             @@
##              dev    #2246       +/-   ##
===========================================
- Coverage   96.17%   86.14%   -10.03%     
===========================================
  Files          30       30               
  Lines        3003     3003               
  Branches      437      437               
===========================================
- Hits         2888     2587      -301     
- Misses         65      324      +259     
- Partials       50       92       +42     
Flag Coverage Δ
integration ?
unit 86.14% <ø> (-0.27%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rly and others added 2 commits August 31, 2026 16:24
…klist

Set the PyNWB 4.2.0 release date to September 1, 2026.

Trim the 4.2.0 changelog entries so each states the user-facing behavior
without recounting the internal cause of the bug.

Add a carve-out to the semantic-versioning checklist item for input that
was accepted but could not be written or read back correctly, and for a
change in the type of an exception that was already raised for the same
input. #2235 falls under both, so 4.2.0 is a minor release.

Indent the continuation lines of steps 2 and 3 in make_a_release.rst so
they render as items of the enumerated list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rly
rly marked this pull request as ready for review September 1, 2026 04:27
The "Run DANDI read tests" workflow points its run step at
tests/read_dandi/read_dandi.py, a path that does not exist, so a
dispatch fails immediately. The workflow is also disabled on a schedule
until dandiset reads can run systematically rather than randomly, which
is tracked by #1804. Reading dandisets is not a release gate until that
is resolved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rly
rly requested a review from oruebel September 1, 2026 04:44
rly and others added 4 commits August 31, 2026 22:46
The example tested for "Chen, Tsai-Wen" while dandiset 000015 stores
"Tsai-Wen Chen", so the else branch fired for every file and the ORCID
reference was never added. Verified by reading the saved archives back:
dandiset_resources.zip now holds 10 rows (5 species + 5 experimenter)
over 2 entities instead of 5 rows over 1 entity, which also exercises
the entity reuse the surrounding text describes.

Stream the first 5 assets rather than all 210. The loop pulled 3.2 GB
from the DANDI Archive on every nightly run to read two metadata strings
per file, and 5 files show a HERD spanning several files just as well.

Add timeout-minutes to the streaming job. On run 33469955713 it hung for
over two hours on a step that normally takes four minutes, because the
job had no timeout and fell back to the six-hour default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conda env update runs the "pip:" block of an environment file as a
sub-process and buffers its output, so when that install stalls the log
holds nothing but a spinner. On run 33474958087 both Windows ros3 jobs
sat in "Installing pip dependencies" for 18 minutes and were cancelled
by the job timeout, with no indication of which package was responsible.

Install remfile and dandi with an explicit "pip install -v" step so pip
reports what it is downloading and building.

Raise the ros3 job timeout to 30 minutes so a slow-but-finite install
completes and reports its duration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The conda solve for "python==3.14" can land on either the GIL or the
free-threaded interpreter. On run 33477855932 the Windows ros3 jobs got
the free-threaded one: every wheel in the log is tagged cp314t and the
build directories are lib.win-amd64-cpython-314t. tensorstore,
numcodecs, pycryptodomex, and google-crc32c publish no cp314t Windows
wheels, so pip compiled all four from source. The first three take about
two minutes between them; tensorstore is a Bazel build that reached
5,051 of 8,228 compile actions before the job timeout cancelled it, and
had been running 24 minutes at that point. Requesting python-gil makes
the choice explicit so the wheels are used.

This reverts commit c9e3e58, whose verbose pip step identified the
source builds. remfile and dandi go back in the environment file so a
developer creating the ros3 environment from it gets a complete
environment.

conda-forge does package dandi for every platform the ros3 jobs use,
including osx-arm64, but only up to 0.75.1, so the pin stays on PyPI.
remfile is not on conda-forge at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rly

rly commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@oruebel This is now good to go

rly and others added 5 commits September 2, 2026 07:33
The comment credited the numcodecs<0.16 cap to dandi, but dandi 0.77.0
and 0.78.0 both require only numcodecs>=0.11.0, and the zarr<=3.1.5 it
requires resolves to numcodecs>=0.14. The cap is pynwb's own, at
pyproject.toml:50, where the zarr extra holds numcodecs below 0.16.0 for
compatibility with zarr<3.

The consequence stands: numcodecs 0.15.1 publishes cp311, cp312, and
cp313 macOS wheels only, while 0.16.5 does publish cp314 ones. So no
dandi release lifts this pin; it moves when the zarr extra does, which
the release checklist ties to hdmf-zarr's bounds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conda-forge published dandi 0.78.0 with cp314 builds for linux-64,
osx-64, osx-arm64, and win-64, covering every platform the ros3 jobs
run on, so the claim that PyPI carries releases newer than the
conda-forge build no longer holds. Drop it; the pin staying on PyPI is
now a choice rather than a constraint, and moving the whole dandi
dependency tree to conda belongs in its own PR against dev.

Name tensorstore, numcodecs, pycryptodomex, and google-crc32c in the
python-gil comment. Release PRs are squash-merged and GitHub keeps only
the title in the squashed commit body, so after this PR merges the
detail behind "See #2246" lives solely in the PR's commit list. The
comment carries enough to re-verify the claim without it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
conda-forge publishes dandi 0.78.0 with cp314 builds for linux-64,
osx-64, osx-arm64, and win-64, covering every platform the ros3 jobs run
on. Taking it from conda lets the solver resolve the dandi tree together
with the rest of the environment, so pycryptodomex, google-crc32c, and
(off Windows) tensorstore arrive as prebuilt conda packages.

Verified with "conda env create --dry-run" for all three subdirs. Each
solves, honours every pin, and selects the GIL interpreter.

Narrow the python-gil comment to tensorstore. pycryptodomex and
google-crc32c now come from conda, and numcodecs 0.15.1, which the zarr
extra caps to, publishes no cp314 wheels at all, so the GIL choice does
not decide whether it compiles. tensorstore is the package the choice
still governs: conda-forge has no win-64 build, so pip supplies it there
from a cp314 wheel that has no free-threaded counterpart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every package in the environment resolves from conda-forge on osx-arm64,
linux-64, and win-64, so defaults contributes nothing to the solve. Both
ros3 workflows already pass conda-remove-defaults: "true" along with
channels: conda-forge to setup-miniconda, so listing it affected only
local environment creation, where it widened the solve to a channel
carrying Anaconda's commercial terms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The conda-forge dandi package requires pynwb, so creating the
environment installs a released pynwb alongside it. The "pip install ."
step in the ros3 jobs replaces that with the working tree build, but the
"conda list" those jobs run for reporting still names the conda-forge
version, which is worth knowing when reading a ros3 log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant