test(xnet): exercise silently dropped responses on subnet deletion#10793
Draft
mraszyk wants to merge 4 commits into
Draft
test(xnet): exercise silently dropped responses on subnet deletion#10793mraszyk wants to merge 4 commits into
mraszyk wants to merge 4 commits into
Conversation
The subnet deletion XNet test only exercised that requests towards a
deleted subnet are rejected, not that responses towards it are silently
dropped (a response can never be rejected).
Add a universal-canister op `LoopUntilGlobalDataSet` that loops making a
self `canister_status` inter-canister call until the canister's global
data equals a trigger blob, then replies with a reply blob. This delays a
reply across message boundaries until an external message sets the global
data.
Use it in `subnet_delete_test` so UC (on the deleted subnet C) makes two
fire-and-forget calls to two canisters on the surviving sender subnet S,
whose replies are released at different times:
- one before US's 2 MB requests, so its response is inducted into the
S->C stream;
- one after the S->C stream is full, so its response stays in the sender's
output queue.
Both responses must be silently dropped when C is deleted. The queued
response is verified via the
`mr_routed_message_count{type="response",status="canister_not_found"}`
metric; the streamed response is dropped together with the whole stream
(intentionally metric-silent), so it is covered indirectly by the
unallowed-`panicked` log guard plus a post-drop health check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t-delete test
Port the response-dropping extension of the system test
`rs/tests/message_routing/xnet/subnet_delete_test.rs` to its StateMachine
analogue.
UC (on the deleted subnet C) makes two fire-and-forget looping calls (the
new `LoopUntilGlobalDataSet` op) to two canisters on the surviving sender
subnet S; their replies are released at different times so that one response
ends up in the S->C stream and the other stays in the sender's output queue.
After C is deleted both must be silently dropped.
Thanks to direct `ReplicatedState` access, the drops are asserted precisely:
the queued response via
`mr_routed_message_count{type="response",status="canister_not_found"} == 1`,
and the streamed response via S's stream towards C being discarded (the
metric-silent bulk drop). Bump the best-effort call timeout so the
early-created responses survive until C is deleted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-delete-response-drop # Conflicts: # Cargo.lock
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The subnet-deletion XNet tests only checked that in-flight requests towards a deleted subnet are rejected. This PR also exercises that in-flight responses towards a deleted subnet are silently dropped (a response can never be turned into a reject).
LoopUntilGlobalDataSetthat loops oncanister_statuscalls (requesting its own status) until its global data is set to a fixed blob, then replies — used to delay a response until a chosen moment.rs/tests/message_routing/xnet/subnet_delete_test.rs) and itsStateMachineanalogue (rs/state_machine_tests/tests/subnet_delete.rs): the deleted subnet's canister calls two canisters on a surviving subnet whose replies are released so that one response ends up in the stream and the other in the sender's output queue when the subnet is deleted.mr_routed_message_count{type="response",status="canister_not_found"}; the streamed response via the stream being discarded (metric-silent bulk drop).