Support `paginationStartKey` parameter in the `archive_v1_storage` JSON-RPC API

[sinzii]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Request

Per the latest archive_v1_storage specification, the paginationStartKey parameter enables specifying a starting key for fetching storage items, which facilitates paginated retrieval.

It appears that the current implementation of archive_v1_storage does not yet support this parameter. Do we have plans to add support for it to ensure the API aligns with the specification?

polkadot-sdk/substrate/client/rpc-spec-v2/src/archive/archive.rs

Lines 217 to 266 in fd2bfd5

	fn archive_v1_storage(
	&self,
	pending: PendingSubscriptionSink,
	hash: Block::Hash,
	items: Vec<StorageQuery<String>>,
	child_trie: Option<String>,
	) {
	let mut storage_client =
	StorageSubscriptionClient::<Client, Block, BE>::new(self.client.clone());
	let fut = async move {
	let Ok(mut sink) = pending.accept().await.map(Subscription::from) else { return };
	let items = match items
	.into_iter()
	.map(|query| {
	let key = StorageKey(parse_hex_param(query.key)?);
	Ok(StorageQuery { key, query_type: query.query_type })
	})
	.collect::<Result<Vec<_>, ArchiveError>>()
	{
	Ok(items) => items,
	Err(error) => {
	let _ = sink.send(&ArchiveStorageEvent::err(error.to_string()));
	return
	},
	};
	let child_trie = child_trie.map(|child_trie| parse_hex_param(child_trie)).transpose();
	let child_trie = match child_trie {
	Ok(child_trie) => child_trie.map(ChildInfo::new_default_from_vec),
	Err(error) => {
	let _ = sink.send(&ArchiveStorageEvent::err(error.to_string()));
	return
	},
	};
	let (tx, mut rx) = tokio::sync::mpsc::channel(STORAGE_QUERY_BUF);
	let storage_fut = storage_client.generate_events(hash, items, child_trie, tx);
	// We don't care about the return value of this join:
	// - process_events might encounter an error (if the client disconnected)
	// - storage_fut might encounter an error while processing a trie queries and
	// the error is propagated via the sink.
	let _ = futures::future::join(storage_fut, process_storage_events(&mut rx, &mut sink))
	.await;
	};
	self.executor.spawn("substrate-rpc-subscription", Some("rpc"), fut.boxed());
	}

[bkchr]

Yeah looks like it is missing. @lexnv do you want to implement it? Otherwise it is probably a good first issue.

[lexnv]

@sinzii Let me know if you'd like to tackle this issue, I could provide some guidance along the way if you are interested 🙏

[sinzii]

@lexnv Yes! I'd be more than happy to tackle this, can you assign it to me.

Let me know if you have any recommendation or suggestion to start this. I see that the this.client.query_iter_pagination_with_producer method here already support query with pagination so I imagine we can just pass along the pagination_start_key to the query to enable this. Is that a good way to start this? Thanks!

polkadot-sdk/substrate/client/rpc-spec-v2/src/common/storage.rs

Lines 276 to 300 in 4e7c968

	StorageQueryType::DescendantsValues => {
	let query = QueryIter {
	query_key: item.key,
	ty: IterQueryType::Value,
	pagination_start_key: None,
	};
	this.client.query_iter_pagination_with_producer(
	query,
	hash,
	child_key.as_ref(),
	&tx,
	)
	},
	StorageQueryType::DescendantsHashes => {
	let query = QueryIter {
	query_key: item.key,
	ty: IterQueryType::Hash,
	pagination_start_key: None,
	};
	this.client.query_iter_pagination_with_producer(
	query,
	hash,
	child_key.as_ref(),
	&tx,
	)

[lexnv]

Yep exactly! Offhand, I would start by introducing the paginationStartKey at:

polkadot-sdk/substrate/client/rpc-spec-v2/src/common/events.rs

Lines 26 to 32 in 41bc46f

	pub struct StorageQuery<Key> {
	/// The provided key.
	pub key: Key,
	/// The type of the storage query.
	#[serde(rename = "type")]
	pub query_type: StorageQueryType,
	}

       // The generic param here would be set to the concrete `StorageKey` while parsing the provided params
       pub pagination_start_key: Option<Key>,

From that point, we'd need to ensure the provided key is valid:

polkadot-sdk/substrate/client/rpc-spec-v2/src/archive/archive.rs

Lines 233 to 234 in 4e7c968

	let key = StorageKey(parse_hex_param(query.key)?);
	Ok(StorageQuery { key, query_type: query.query_type })

• A similar check to let key = StorageKey(parse_hex_param(query.key)?); could be added for the pagination key
• The pagination key should be Some(_) only if the this is a StorageQueryType::DescendantsValues or StorageQueryType::DescendantsHashes

Then it remains to utilize the key in the StorageSubscriptionClient code 🙏