manifest v2 for non co-located seg and meshes#542
Open
akhileshh wants to merge 8 commits into
Open
Conversation
MeshMeta wraps a ChunkedGraph and exposes mesh dir locations and per-dataset params from custom_data["mesh"] (the single source of truth), shared branch-agnostically across the pcgv2/pcgv3 lines. Resolves absolute initial/dynamic dirs so meshes can live in a different bucket than the watershed, and flags graphs old clients can't reach. Co-Authored-By: Claude <noreply@anthropic.com>
Replace scattered custom_data["mesh"] / dataset_info reads (mesh dir, max_layer/mip/max_error, initial_ts, dynamic path) with MeshMeta. Removes the duplicated remesh meta blocks and dead unsharded computes, and realigns remeshing onto custom_data as the source of truth. Co-Authored-By: Claude <noreply@anthropic.com>
Add a graph.basetypes compat shim and point manifest imports at it, switch np.in1d->np.isin (in1d is gone in numpy 2.x), and adopt pcgv3's cache TTL + clear_namespace. Converges the meshing manifest across the pcgv2/pcgv3 lines so v2 work cherry-picks clean. Co-Authored-By: Claude <noreply@anthropic.com>
Branch-agnostic manifest/v2.py: parse the client's advertised manifest version from the Accept header, and assemble the v2 format that groups fragments by absolute mesh bucket path so mesh location travels in the manifest itself. Co-Authored-By: Claude <noreply@anthropic.com>
Serve the v2 grouped-by-bucket manifest to clients that advertise manifest_version=2, else the v1 flat list (byte-identical). Refuse with 406 when a migrated graph (meshes at an absolute path) is requested without v2, so old clients fail loudly instead of getting unreachable paths. Co-Authored-By: Claude <noreply@anthropic.com>
Anchor the verify=1 shard reader at initial_path (via reader_anchor) so migrated datasets read initial meshes from the real bucket, and append Accept-Encoding to Vary so manifests keep advertising Accept. Co-Authored-By: Claude <noreply@anthropic.com>
v2 emits each fragment identical to v1 (seg id prepended when asked) and honors return_seg_ids, differing only by grouping fragments under their bucket key, so the client reuses its v1 handling unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
Emit initial/dynamic absolute bucket locations in mesh_metadata on both the served /info endpoint and get_json_info, so external clients and PCG's own manifest-bypass reads resolve meshes from the real buckets. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.