-
Notifications
You must be signed in to change notification settings - Fork 40
Open
Labels
team/fs-wgItems being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10Items being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10
Milestone
Description
Summary
Add a POST /pdp/piece/fetch endpoint allowing SPs to fetch pieces from other SPs via HTTP, enabling multi-copy uploads where the client uploads once and additional SPs pull from the first.
Motivation
Synapse SDK needs to store pieces on multiple SPs for redundancy. Currently this requires parallel uploads of the same data. For large pieces (up to 1 GiB), this is inefficient and problematic for streaming APIs.
With this feature:
- Client uploads to endorsed SP
- Client requests second SP to fetch piece from first SP
- Second SP downloads via existing /piece/{cid} retrieval endpoint
- Client calls AddPieces on both SPs
Additionally this functionality that will be required by the current MSP MVP design (under development). This is also similar to the existing DataSourceHTTP functionality in the current main branch markets implementation and shouldn't over-complicate the merge-back to main process.
Approach
- New endpoint accepts piece CIDs + source URLs +
extraDatablob (for auth gate-keeping: server can eth_call to check the client is able to pay for this) - Endpoint is idempotent, keyed by a hash of
extraData, so can be called many times to retrieve current status - Downloads queued and deduplicated by pieceCid
- Harmony task handles downloads with configurable concurrency and timeouts to protect both source and destination SPs
- CommP verifies download
- Integrates with existing parked_pieces flow, the remainder of PDP onboarding remains the same after piece acquisition
Metadata
Metadata
Assignees
Labels
team/fs-wgItems being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10Items being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10