Add Zarr v3 reader#853
Draft
ehennestad wants to merge 7 commits into
Draft
Conversation
Implements io.backend.zarr3.Zarr3Reader/Zarr3Writer/Zarr3LazyArray on top
of the zarr-matlab package, wired into BackendFactory alongside the
existing HDF5 and Zarr v2 backends.
Key design points:
- Zarr v3 stores may be written by Python tools using numpy/row-major
shape order; rank >= 2 dims and data are reversed on both read and
write to match MatNWB's H5-style convention (matches
io.backend.zarr2.Zarr2Reader/HDF5Reader), via a shared
io.internal.zarr3.normalizeDatasetDimensions helper.
- Object references use the same attribute-value convention as
io.backend.zarr2 / hdmf-zarr: struct('zarr_dtype',"object",'value',...).
A dataset whose elements are themselves references (e.g. a DynamicTable
column of ElectrodeGroup references) is decoded into a
types.untyped.ObjectView object array.
- Compound (struct/table) datasets -- a Zarr v3 "structured" data_type,
e.g. PlaneSegmentation's pixel_mask/voxel_mask or
TimeSeriesReferenceVectorData's response/stimulus columns -- are backed
by Zarr3LazyArray, converting zarr-matlab's array-of-records into the
"struct of arrays"/table shape io.backend.hdf5.@HDF5LazyArray produces,
with reference-tagged fields decoded the same way as above. The
DataStub's compound type descriptor is built to match what
types.util.checkDtype validates against the schema without loading data.
- A hdmf-zarr "scalar" zarr_dtype marker (rank-1, length-1 array
representing a true NWB scalar property) is required to read a dataset
eagerly, rather than element count alone -- a one-row VectorData column
also has Dataspace.Size == 1 but must not be collapsed to a bare value.
Requires a zarr-matlab build with "structured"/"fixed_length_utf32"
Zarr v3 data type support (unstable, unspecified zarr-python extensions)
for the compound-dataset case; verified against real NWB Zarr v3 example
files and byte-for-byte against zarr-python 3.1.6.
Includes Zarr3ReaderTest, Zarr3LazyArrayTest, Zarr3WriterTest, and a
generated-fixture helper mirroring the Zarr v2 test suite's structure.
The reader/lazy-array support for a "structured" data_type was already implemented (pixel_mask-style compound datasets, and reference-tagged compound fields), but the shared test fixture and reader/lazy-array test suites never exercised it, and the fixture docstring still incorrectly claimed compound datasets were unsupported. Adds a 3-record pixel_mask compound array (x uint32, y uint32, weight float32) to the fixture, built via zarr.metadata.ArrayMetadata + a direct zarr.Array construction since zarr.create's public API has no way to pass a "structured" data_type's field configuration yet. Verified against real zarr-python 3.1.6 output and MATLAB execution: full load returns a struct-of-arrays (matching the HDF5 backend convention), indexed access returns a table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add the MatBox install-matbox action and a matbox.installRequirements('.')
call (reading requirements.txt) to the main test workflow, so the Zarr v3
reader tests run against a real zarr-matlab install instead of skipping.
Also add a dedicated run-zarr-tests.yml workflow scoped to Zarr-related
paths, and ignore the external_packages/zarr-matlab install location.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not needed; Zarr tests are covered by the main test workflow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ehennestad
force-pushed
the
zarr-support/5-zarr3-reader
branch
from
a7c904d to
4d725c0
Compare
getZarr3MatlabPath only checked ZARR3_MATLAB_PATH and
external_packages/zarr-matlab, so tests skipped silently when
matbox.installRequirements put the package elsewhere (its default
add-ons folder) and added it to the path directly. Resolving via
which("zarr.open") first, matching the check already used by
io.backend.zarr3.internal.ensureAvailable, finds the package
regardless of how it was installed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Add reader for Zarr v3 files.
Needs: https://github.com/catalystneuro/zarr-matlab
How to test the behavior?
Checklist
fix #XXwhereXXis the issue number?