Add hdmf.typing: replace docval with type hints (Phase 1-2 of #1129)#1526
Draft
bendichter wants to merge 8 commits into
Draft
Add hdmf.typing: replace docval with type hints (Phase 1-2 of #1129)#1526bendichter wants to merge 8 commits into
bendichter wants to merge 8 commits into
Conversation
…1129) Adds the hdmf.typing package, the first phase of replacing @DocVal with standard Python type hints while keeping downstream libraries working: - Type aliases with docval-equivalent semantics: Int/UInt/Float/Bool (numpy widening), ArrayData/ScalarData/AnyData (live macro registry), TypeName[...] (MRO-name forward refs), Shaped[...] (array shapes) - @validated decorator: docval-parity runtime validation of type-hinted functions (same error messages, TermSetWrapper unwrapping, allow_positional, HDMF_TYPE_CHECKING=off kill switch); hints docval cannot express are checked via beartype/numpydantic - get_docval now synthesizes docval-compatible argument specs from signature + type hints + Google-style docstring for functions without __docval__, so the downstream splice pattern @DocVal(*get_docval(Parent.__init__, ...)) keeps working as modules migrate off @DocVal - Migration tool (python -m hdmf.typing.migrate) and parity-testing helpers (hdmf.typing.testing) - New dependencies: beartype, numpydantic, docstring_parser Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-only Rearchitect hdmf.typing per review: validation runs purely through the type-hint system, with the docval-spec synthesis kept only as a backwards-compatibility export for get_docval (to be removed with docval). - Macro aliases (ArrayData/ScalarData/AnyData), TypeName[...], and Shaped[...] now carry real beartype.vale.Is validators, so plain @beartype and beartype.door.is_bearable enforce them anywhere; compat info for the get_docval shim lives in a side registry keyed by validator identity (BeartypeValidator is slotted) - Numeric aliases (Int/UInt/Float/Bool) are plain unions checked natively by beartype; the shim collapses complete alias sets in synthesized specs back to docval's 'int'/'uint'/'float'/'bool' vocabulary - @validated checks every parameter via is_bearable on the original hint (numpydantic NDArray hints via numpydantic's isinstance); no more routing through hdmf.utils.check_type. Bare int/float/bool hints are now strict (use the aliases for numpy widening) - @validated strips shape validators from the hint for its type check so shape violations keep raising ValueError with docval's message (its own shape pass also implements the unwrap-by-argname fallback beartype cannot express); the full annotation still enforces shape under plain beartype - Added hdmf.typing.register_macro as the docval_macro successor - New tests pin the architecture: aliases enforced by plain beartype, bare-int strictness, numeric-union collapse in synthesized specs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tart) First real-module migration for #1129. All 12 @DocVal sites in hdmf.data_utils now use real type-hinted signatures: - GenericDataChunkIterator, DataChunkIterator (+from_iterable), DataChunk, DataIO, ShapeValidatorResult constructors use @validated - recommended_chunk_shape/recommended_data_shape drop their decorator (docval was only generating their docstrings) - The __docval_init class-attribute spec tuples are gone; downstream introspection via get_docval() now serves synthesized specs Verified: full hdmf suite green; pynwb dev suite (804 tests, 6581 subtests) green against this branch in a clean venv, including the H5DataIO splice @DocVal(*get_docval(DataIO.__init__), ...) now running on synthesized specs, and iterative-write roundtrips through the migrated DataChunkIterator. Migration tool hardening from this exercise: dict(...) call specs, attribute type expressions (np.ndarray), docval type None -> Any, dotted-target getargs rewrite (self.x, self.y = getargs(...)), existing return annotations preserved, non-literal returns docs and leftover kwargs references flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #1526 +/- ##
==========================================
- Coverage 93.22% 91.79% -1.44%
==========================================
Files 41 50 +9
Lines 10168 11027 +859
Branches 2100 2302 +202
==========================================
+ Hits 9479 10122 +643
- Misses 413 558 +145
- Partials 276 347 +71 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Migrates all @DocVal sites in query.py (HDMFDataset), common/sparse.py (CSRMatrix), common/hierarchicaltable.py, validate/validator.py (all validator classes), and common/resources.py (HERD and rows) to type-hinted signatures with @validated. No @DocVal usage remains in these modules (only @docval_macro registrations). Migration tool upgrades from this batch: - logical-line joining, so multi-line getargs/popargs statements (parenthesized or backslash-continued) are rewritten too - `x = kwargs['x']` subscript idiom rewritten/removed - trailing commas in getargs(...) calls handled Hand-finished: super().__init__(**kwargs) -> explicit keywords (4 identical validator sites, CSRMatrix, HERD), docs that the original specs spelled as accidental string tuples, and docstring titles. Verified: full hdmf suite green (1984); full pynwb dev suite green (804 tests + 6581 subtests) against this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The migrate tool replaced source lines starting at the @DocVal decorator but re-emitted ALL decorators, leaving originals above @DocVal (e.g. @classmethod, @AbstractMethod) duplicated. classmethod(classmethod(f)) relied on descriptor chaining that Python 3.13 removed, so HERD.from_zip and get_zip_directory raised "TypeError: 'classmethod' object is not callable" on 3.13/3.14 — the cause of all failing CI jobs (3.13/3.14 test matrices, sphinx-links, readthedocs gallery). - migrate tool now replaces from the first decorator line - dedupe the three affected sites (HERD.from_zip, HERD.get_zip_directory, Validator.validate) and add a regression test for decorator emission Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
autodoc expands the hdmf.typing numeric alias unions into signatures and cannot resolve numpy scalar types (or pandas internal class paths) as py:class intersphinx targets; ignore those by regex. Also turn cross-references to undocumented targets (module attributes and the private MappedHint class) into code literals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The validator checker closures all rendered identically in autodoc signatures (Is[macro_validator.checker]) and produced unresolvable cross-reference warnings under the linkcheck builder. Give each checker a descriptive __qualname__ (is_array_data, is_DynamicTable, has_shape_anyx3, ...) so alias signatures are distinct and readable, ignore the remaining beartype-internal xref targets, and drop PEP 563 annotations from hdmf.typing.testing so autodoc resolves them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stringified dataclass annotations (dict[str, str]) produced the last unresolvable-xref warning in the docs build; all annotations in the module are valid at runtime on Python 3.10+. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
First implementation phases of #1129: replace
@docvalwith standard Python type hints, without breaking downstream libraries (PyNWB alone has ~119get_docvalsplice sites).Architecture: validation runs purely through the type-hint system. Type checks go through beartype and array dtype/shape checks through numpydantic. docval is untouched and remains fully functional; the only docval-facing piece of the new package is a compat layer that synthesizes docval-format argument specs from type-hinted signatures, so
get_docval()keeps working on migrated functions. That layer is removed together with docval, eventually.What this adds (
hdmf.typing)@beartype/is_bearableanywhere:Int,UInt,Float,Bool— accept numpy scalar types (a bareinthint is strict; useIntfor numpy widening)ArrayData,ScalarData,AnyData— backed by the live macro registry (hdmf-zarr registrations still apply)TypeName['DynamicTable']— MRO-name matching for cross-module forward referencesShaped[ArrayData, (None, 3)]— docval-style shape specs; numpydanticNDArray[Shape[...], dtype]also supported@validated— call-time validation of type-hinted functions: docval-format error messages,TermSetWrapperunwrapping, shape-unwrap-by-argname fallback,allow_positionalparity,HDMF_TYPE_CHECKING=offkill switchget_docvalfallback (~15 lines inutils.py): synthesizes docval-compatible specs from signature + hints + Google-style docstring for functions without__docval__. Downstream@docval(*get_docval(Parent.__init__, ...))splices keep working unchanged.python -m hdmf.typing.migrate file.py --diff|--in-place(AST-based; flags anything unsafe with# TODO(migrate))hdmf.typing.testingfor migration PRs here and downstreamMigrated modules
data_utils.py,query.py,validate/validator.py,common/sparse.py,common/resources.py(HERD), andcommon/hierarchicaltable.pynow use type-hinted signatures throughout. This exercises the riskiest compat path in production:H5DataIOdecorates itself with@docval(*get_docval(DataIO.__init__), ...), which now runs on synthesized specs.Verification
test_docval.pyretained verbatim@docvalmust accept/reject identically),@validatedsemantics, beartype-native alias enforcement, migration toolDataChunkIteratorand theH5DataIOspliceRollout plan (from #1129 planning)
hdmf.typing, docval untoucheddata_utils,query,validate/validator,common/sparse,common/resources,common/hierarchicaltableare fully migrated (region/monitor/arrayno longer exist on dev). ~52 docval sites removed; the full PyNWB dev suite (804 tests + 6581 subtests) passes against every commit of this branch.container.py,common/table.py,spec/,build/), classgenerator emitting real signatures,py.typed, docval deprecation much laterNew required dependencies:
beartype,numpydantic,docstring_parser.🤖 Generated with Claude Code