Skip to content

Add experimental Pyxa (Stellaromics) reader - #425

Open
ckmah wants to merge 37 commits into
scverse:mainfrom
ckmah:pyxa-reader
Open

ckmah wants to merge 37 commits into
scverse:mainfrom
ckmah:pyxa-reader

Conversation

@ckmah

@ckmah ckmah commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Adds an experimental reader for Stellaromics Pyxa output, spatialdata_io.experimental.pyxa(), plus a pyxa CLI command.

What it reads

File Element
cell_by_gene_v1.csv + cell_metadata_v1.csv (required) rna table: sparse counts in X, obsm["spatial"] in µm
pyxa_studio_v1.csv Cluster (categorical) and obsm["X_umap"] on the rna table; cells Pyxa Studio filtered out keep missing values
cell_assigned_gene_v1.csv transcripts (3D points; unassigned transcripts, with a cell_id ending in _-1, are kept and flagged by an assigned column)
segmentation_geometries_v1.parquet cell_boundaries (one footprint per cell, annotated by rna) and cell_boundaries_z (one polygon per cell per z-plane)
mosaic_3d.ome.zarr, or mosaic_3d.ome.zarr.zip read in place mosaic_image, multiscale, using the pyramid levels already in the store
segmentation_geometries_v1.parquet + mosaic, with labels=True cell_labels: 3D labels on the mosaic's voxel grid (same levels and transform as mosaic_image), annotated by rna

Each optional file is read when it is in path. Passing False skips it, True requires it, and a path reads that file from elsewhere. image= works the same way for the mosaic, directory or zip. shapes= controls the shapes: by default they are returned when the polygons are read and labels is False.

Every element is in µm in the global coordinate system:

  • The segmentation polygons are stored in pixels. The reader scales them to µm, then repairs and checks the ones that the float scaling makes self-intersecting.
  • Their z-plane centre is added as a Z_um column, since shapes are 2D in spatialdata.
  • The xy and z voxel sizes are fitted from a sample of cell_metadata rows, where each cell centroid is given in both pixels and µm. The reader raises an error if the two aren't related by a pure scale.

Each cell's centroid in cell_metadata is exactly the area-weighted centroid of its polygons, which the tests check in x, y and z.

Cells stacked in z have overlapping footprints. cell_boundaries is therefore meant for 2D display and table annotation, not for 2D spatial aggregation.

3D cell labels (labels=True)

The polygons are rasterized onto the mosaic's grid, so labels and image overlay voxel for voxel. The table then annotates cell_labels through an integer label_id: the trailing integer of cell_id (Region_17 → 17) when those are unique, positive and below 2³¹, otherwise 1..n in table order.

  • Lazy. Level 0 is one dask.delayed drawing task per 32 × 1024 × 1024 tile, run when the element is computed or written. Coarser levels are strided (nearest-neighbour) views of level 0, sampled at block centres like the mosaic's own pyramid, with steps taken from the mosaic's OME-NGFF scales. spatialdata writes all levels in one compute, so each tile is drawn once per write.
  • Polygon decoding is eager. It runs one task per parquet row group in joblib worker processes (prefer="processes"; configurable with joblib.parallel_config), streaming 65,536-row batches. The workers are shut down afterwards.
  • Drawing uses Pillow. Where two cells overlap on a plane, the higher label wins. Holes are filled.
  • Scale. On a full region (492k cells, 23.2M polygons, 284 × 9,786 × 11,889 mosaic): reading takes about 1.5 min, writing about 6 min, with a peak of about 30 GB. On a 32 × 1024 × 1024 window, level-0 voxels agreed 99.9994% with an independent rasterizer. That was measured before a tile-seam fix; the fix changed only voxels on tile seams.

Specification

There is no public format specification yet, so the reader is under experimental. It is validated against the public Stellaromics/demo dataset (BSD 3-Clause).

Test data

  • xsmall/ in that dataset is a 100 × 100 × 100 µm crop: 187 cells, about 23k transcripts and the DAPI mosaic, about 8 MB in total.
  • prepare_test_data.yaml downloads it to data/pyxa_xsmall.
  • Until the shared CI artifact is regenerated with pyxa_xsmall (the workflow notes this needs a branch in this repository), tests/test_pyxa.py downloads the same 5 files from the Hub when data/pyxa_xsmall is missing, in a way that is safe with pytest -n auto, and skips the module if the download fails. That fallback can be removed once the artifact has the data.

Scripts to download and convert the dataset are in giovp/spatialdata-sandbox#65 (pyxa_v1_io).

Tests (tests/test_pyxa.py, 75)

  • Example data: data extent, index integrity of every element, the table annotation joins, and the CLI (--labels, --no-shapes, --image/--no-image, --skip).
  • Mosaic: loading from the directory, from a zip (a single top-level entry, the group at the root, or with __MACOSX/ present), default lookup, image=False.
  • Labels:
    • label-id rules and fallbacks;
    • ring decoding (empty parquet, polygons off the z range, cells not in the table, several row groups, batches smaller than a row group);
    • tile planning and drawing (a square, higher label wins, seams at fractional edges and for random rings);
    • laziness, and each level-0 tile drawn once per write;
    • coarse levels exactly equal to level 0 sampled at block centres, with the guard when a level cannot fit;
    • the worker-pool shutdown and parallel_config;
    • a write/read round trip with label_id resolving.
  • Helpers: column validation, voxel-size fit, geometry repair, footprint union, multiscale image loading, µm conversion, sparse counts.

Checklist

  • Reader under experimental/; string constants in PyxaKeys
  • Small public test dataset with a permissive license (BSD 3-Clause)
  • Extent and index-integrity proxy tests; tests for the helper functions
  • Visual check of the alignment between image, polygons and transcripts
  • CLI command
  • Download and conversion scripts for spatialdata-sandbox
  • 3D labels checked on a full region against an independent rasterizer

Narrows the tests/data gitignore rule so this small (~1.5MB) fixture
can be checked in, since no public host exists for Pyxa data (unlike
other readers' CI-downloaded fixtures).
Assembles points (transcripts, dask-backed), shapes (segmentation),
and table (expression + metadata) into one SpatialData object.
Registers pyxa in spatialdata_io.experimental (not the top-level
package) matching the existing precedent for iss, and adds the
`spatialdata-io pyxa` CLI command.
Not part of this repo's build (hatchling-based, not uv) -- it's local
dev tooling artifact from installing test dependencies with uv.
Adds an image_path parameter to pyxa() that loads a full-resolution
OME-Zarr (OME-NGFF v0.5) mosaic image (e.g. DAPI) into
sdata.images["mosaic_image"], using its coordinateTransformations
metadata so it aligns with the points/shapes in microns. The image is
not colocated with the other 4 Pyxa output files in the real pipeline
layout, so it's a separate optional argument rather than discovered
from `path`. Only the full-resolution level (scale0) is read; reusing
the precomputed pyramid is left for later.
After rebasing onto upstream main: import the reader lazily inside
pyxa_wrapper (upstream dropped the globals() injection in __main__),
expose --image-path so CLI options mirror the reader API, and fix
mypy/ruff findings (typed OME-Zarr attrs, zip(strict=True), docstring).
Reader:
- Convert segmentation polygons from pixels to micrometers in the reader,
  so every element is returned in um with an identity transformation.
  Polygons made invalid by the float scaling are repaired with
  shapely.make_valid (structure method, polygonal parts only) and validated.
- Add a Z_um column to the shapes: the z-plane centre, (ZIndex + 0.5) * dz,
  since shapes are 2D in spatialdata. cell_metadata Z_pixels is exactly the
  area-weighted mean ZIndex + 0.5 of each cell.
- Infer the (xy, z) voxel size from a sample of cell_metadata rows and raise
  if pixel and um centroids are not related by a pure scale.
- Make the gene categories known before PointsModel.parse to avoid its
  unknown-categories warning.
- Remove references to internal documentation and pipeline naming.

Tests:
- Replace the checked-in fixture with the xsmall crop (100 um cube) of the
  public Stellaromics/demo Hugging Face dataset, downloaded in
  prepare_test_data.yaml like the other readers' test data.
- Check that area-weighted polygon centroids match the per-cell metadata
  centroids in x, y and z.
- Return two shapes elements: cell_boundaries, one footprint per cell (the
  union of its z-plane polygons, merged in a thread pool) indexed by cell_id
  and annotated by the rna table; and cell_boundaries_z, the per-plane
  polygons with a unique index and cell_id / ZIndex / Z_um columns. A table
  cannot annotate an element with repeated instance ids: the joins either
  failed or duplicated rows.
- Drop the obs index name that clashed with the cell_id column in joins.
- Load every precomputed level of the mosaic OME-Zarr as a multiscale image
  instead of only the full-resolution level.
- Add example-data tests on the xsmall dataset following the contribution
  guide: data extent, index integrity of every element and of the table
  annotation, and the CLI with the mosaic.
ckmah and others added 13 commits September 25, 2026 14:36
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
…s image_path

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
…cell_id

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
…s= option

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
ckmah and others added 8 commits September 28, 2026 21:13
Co-authored-by: ckmah <clarence.mah@stellaromics.com>
A ring whose high bound lies within half a voxel of a tile edge fills the
next tile's first voxel, where it was never assigned. Tiles are now assigned
from bounds padded by one voxel on the high side.

PIL also truncates float vertices toward zero and mis-draws polygons with
negative vertices, so a ring reaching into a tile from its low side drew
differently tiled than whole. Vertices are now snapped to the nearest voxel
before the tile offset and drawn on a canvas starting at the lowest vertex,
then cropped: every tiling draws exactly the whole-level result. Drops the
dead empty-idx branch.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Counting stand-ins for _rasterize_tile are small defs instead of
`append(...) or real(...)` lambdas, and labels levels are read with
np.asarray(....data) rather than the DataTree-typed `.values`.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Row groups are decoded with joblib.Parallel(prefer="processes") over up to
joblib.cpu_count() workers, so joblib.parallel_config(backend=...) can pick
another backend. When the default loky backend ran, its reusable executor
is shut down after decoding so idle workers do not hold memory through the
write.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Coarse label levels stride level 0 from offset step // 2 per axis (clamped so
the level keeps the mosaic's shape) instead of from 0. Meteor's pyramid is a
smoothed block average centred on each block (ngff-zarr itkwasm_gaussian;
level-i voxel 0 centred at level-0 index (step - 1) / 2): on the uncropped
colon mosaic, centre samples correlate with the stored level at 0.973 /
0.931 / 0.857 (levels 2-4) against 0.933 / 0.839 / 0.744 from offset 0.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
- CHANGELOG: one "Added: experimental pyxa reader" entry (unreleased).
- pyxa docstring: order-of-magnitude decode cost only, one task per row
  group across up to the CPU count of workers, ZipStore scheduler advice,
  shapes/footprint text conditional on labels, and a note that the lazy
  labels' graph holds every ring's coordinates.
- _get_labels logs rings and level-0 tiles alongside levels.
- pillow is a declared dependency (PIL is imported directly).
- _open_mosaic ignores top-level __MACOSX/ (any "__" entry) in a zip.
- CLI: --image with --no-image is a usage error.
- Tests: zip with the group at its root, zip with __MACOSX/, --no-shapes,
  --image/--no-image conflict, the labels log line.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
Move _pyxa_labels.py's contents into pyxa.py as one coherent section
(_MosaicGrid next to _mosaic_grid; the rest after the image helpers and
before pyxa()), merge imports, and delete the now-empty module. Tests
import from spatialdata_io.readers.pyxa and monkeypatch that module
instead of the old _pyxa_labels one.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
_MosaicGrid.step() rounded the ratio of level shapes, which drifts up to
one coarse plane from the image whenever a level's array shape isn't an
exact multiple of level 0's (e.g. the colon mosaic: z 284/17 = 16.7 rounds
to 17, but the OME scale ratio is exactly 16). _MosaicGrid now carries
every level's OME-NGFF scale, and step() derives the stride from the
scale ratio to level 0 instead, raising if a ratio isn't within 1e-6 of a
positive integer.

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
…ypy-clean

Co-authored-by: ckmah <clarence.mah@stellaromics.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.53788% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.42%. Comparing base (261578f) to head (9a85304).

Files with missing lines Patch % Lines
src/spatialdata_io/readers/pyxa.py 97.23% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #425      +/-   ##
==========================================
+ Coverage   63.53%   68.42%   +4.89%     
==========================================
  Files          26       27       +1     
  Lines        3263     3791     +528     
==========================================
+ Hits         2073     2594     +521     
- Misses       1190     1197       +7     
Files with missing lines Coverage Δ
src/spatialdata_io/__main__.py 85.95% <100.00%> (+1.40%) ⬆️
src/spatialdata_io/_constants/_constants.py 100.00% <100.00%> (ø)
src/spatialdata_io/experimental/__init__.py 100.00% <100.00%> (+100.00%) ⬆️
src/spatialdata_io/readers/pyxa.py 97.23% <97.23%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ckmah
ckmah marked this pull request as ready for review October 2, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants