[stevevanhooser]

Problem

In the lightsheet viewer, tiles drew slowly in two ways:

1. Tiles already on disk. The status showed "Loading view 50/112" with nothing in flight. Both viewers built each pyramid level from one da.from_delayed array per chunk or tile, joined with da.block. That gives the graph one layer per chunk, and dask walks every layer each time napari computes a slice. So the cost grew with the size of the whole level, not with the tiles on screen.
2. Downloaded tiles. The timing log added here showed each new tile taking 30–85 s in total, against 0.2–7 s of transfer. The rest was spent adding the tile to DID's file cache:
   • With tens of thousands of cached files and dozens of download threads finishing together, each is_file / add_file scanned the cache index row by row.
   • DID-python's lock either let a second thread through unguarded or made it sleep a whole second per retry.
   • A lock left behind by a killed viewer blocked every later add for an hour (Python's expiry), 30 s per tile.

Benchmark for (1): a view of 16 chunks, all on local disk, chunks 64×256×256 uint16.

Chunks in level	Before	After
256	0.05 s	0.02 s
16,384	1.4 s	0.02 s
131,072	26 s	0.02 s

Reading the 16 files directly takes about 15 ms. Building a 131k-chunk level at open time also dropped from about 38 s to 2 s.

Changes

• ndi.pyramid.multiscale: _build_block_grid is replaced by _build_level_array. Each level is now a single da.map_blocks layer, and each block works out its chunk from block_info. The stored short-circuit for sparse chunks, edge-block shapes and the upsample-fallback reader all work as before.
• Tile timing log: after each burst of reads, the viewer prints one [lightsheet] tiles: line to stderr. It splits tile time into path lookup (memo hit or the fetcher queue, plus how many of those the workers found on disk vs. downloaded), disk read and decode, and gives the burst's wall-clock span. NDI_LIGHTSHEET_TIMING=0 turns it off. With NDI_LIGHTSHEET_DEBUG, the per-fetch line also reports cache add Xs.
• Storing fetched tiles (_ChunkFetcher._keepFetched): checks the disk (one stat) instead of the cache index, and lets one fetcher thread at a time add to DID's cache. The underlying cache fixes are in DID itself, FileCache: thread-safe lock, binary-search lookups, cache lock handed on quickly and expiring like MATLAB's VH-Lab/DID-python#89 and fileCache: binary-search the index; the cache lock is handed on quickly VH-Lab/DID-matlab#221, both merged.
• Download connection pool (ndi.cloud.api.files._download_session): pools 64 connections per host, one per possible download thread. requests' default of 10 made every download past the tenth open and discard its own connection ("Connection pool is full, discarding connection").
• ndi.gui.app.genepyramid.multiscale.levelArrays: the same one-layer-per-level fix. Levels are still cropped to their true size.
• CHANGELOG entry under Unreleased → Fixed.

Testing

• Tests touching the pyramid, both viewers, cloud downloads, bulk upload and ctest pass after merging main: 909 passed, 196 skipped. The gene-pyramid tests include test_a_computed_block_matches_readViewport and test_only_the_touched_tiles_are_fetched.
• New tests:
  • 16 threads adding fetched members to one real FileCache all land without touching is_file. This passes against DID-python both before and after Navigator: statusIcon, progressPane, sessionInfo #89, and fails without this change.
  • A member that's already cached isn't added twice.
  • The download session pools at least MAX_WORKERS connections.
• Checked by hand that a ragged, sparse level gives the right values and edge shapes. ruff and black are clean.
• CI on 1dde67f, after merging main with Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046) #340: everything passes, including bridge completeness and all four cloud jobs (the pyramid integration tests now read Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046) #340's rebuilt dataset). At the time of writing only the two Windows smoke tests were still running.
• Run in napari on a real dataset on an M4 Mac mini: views of cached tiles now fill in without the pause, and new tiles arrive at network speed once the stale cache lock is gone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJ9PUeM7x2vLsg8FcZQfXo

[stevevanhooser]

Two checks are red on this PR, and neither comes from this change:

• MATLAB/Python bridge completeness: +ndi/+cloud/+api/+documents/waitForDocumentCount.m was added to NDI-matlab main (4129d1b) after main here last ran CI green, and the check reads NDI-matlab main at run time. Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046) #340 ports it. That PR is a larger feature (bulk document upload), so I haven't copied it into this one. A placeholder bridge entry here would conflict with Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046) #340's real entry. This check goes green once Bulk document upload with ingestion wait (NDI-matlab#1047); ctest failure values (NDI-matlab#1046) #340 merges and this branch picks it up.
• Cloud API (User 1, prod): test_pyramid_loader_integration.py::TestLoaderReadsRealBytes::test_the_coarsest_level_computes_and_is_not_all_zero fails the same way on main, in the scheduled runs of Oct 21, 22 and 23 (e.g. run 39894603671), before this change. No fix exists for it yet.

Every other check that has finished is green, including test (3.10/3.11/3.12) and lint.

---

Generated by Claude Code