Skip to content

pdpv0: Add SP-to-SP piece fetch endpoint #828

@rvagg

Description

@rvagg

Summary

Add a POST /pdp/piece/fetch endpoint allowing SPs to fetch pieces from other SPs via HTTP, enabling multi-copy uploads where the client uploads once and additional SPs pull from the first.

Motivation

Synapse SDK needs to store pieces on multiple SPs for redundancy. Currently this requires parallel uploads of the same data. For large pieces (up to 1 GiB), this is inefficient and problematic for streaming APIs.

With this feature:

  1. Client uploads to endorsed SP
  2. Client requests second SP to fetch piece from first SP
  3. Second SP downloads via existing /piece/{cid} retrieval endpoint
  4. Client calls AddPieces on both SPs

Additionally this functionality that will be required by the current MSP MVP design (under development). This is also similar to the existing DataSourceHTTP functionality in the current main branch markets implementation and shouldn't over-complicate the merge-back to main process.

Approach

  • New endpoint accepts piece CIDs + source URLs + extraData blob (for auth gate-keeping: server can eth_call to check the client is able to pay for this)
  • Endpoint is idempotent, keyed by a hash of extraData, so can be called many times to retrieve current status
  • Downloads queued and deduplicated by pieceCid
  • Harmony task handles downloads with configurable concurrency and timeouts to protect both source and destination SPs
  • CommP verifies download
  • Integrates with existing parked_pieces flow, the remainder of PDP onboarding remains the same after piece acquisition

Metadata

Metadata

Assignees

Labels

team/fs-wgItems being worked on or tracked by the "FS Working Group". See FilOzone/github-mgmt #10

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions