fix: deflake //rs/bitcoin/adapter:adapter_integration_test#10765
Open
basvandijk wants to merge 5 commits into
Open
fix: deflake //rs/bitcoin/adapter:adapter_integration_test#10765basvandijk wants to merge 5 commits into
basvandijk wants to merge 5 commits into
Conversation
The doge_test_send_tx sub-test flakes because the adapter's transaction relay is a one-shot: a transaction is advertised (inv) to each peer exactly once, the mark is set even if the send fails, and it survives a connection discard + reconnect. If any step of the unacknowledged inv -> getdata -> tx handshake is lost (connection churn / CPU contention on CI), the transaction is silently never relayed and Bob's balance stays 0. Fix: track when each peer was last advertised to and re-advertise every TX_READVERTISE_PERIOD_SECS (10s) for as long as the transaction stays in the cache. Peers that already received the transaction simply ignore the repeated inv. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR deflakes //rs/bitcoin/adapter:adapter_integration_test by making the adapter’s transaction relay resilient to dropped inv → getdata → tx handshakes, via periodic re-advertisement while a transaction remains cached.
Changes:
- Change per-peer transaction advertisement tracking from a one-shot set to a timestamped map, and re-advertise after
TX_READVERTISE_PERIOD_SECS(10s). - Fix an inventory overflow-path bug where a partially-built
invcould be broadcast to the wrong peers. - Update/add unit tests to validate “no re-advertise within period” and “re-advertise after period” behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Addresses Copilot review comments: the implementation checks `last_advertised + period <= now`, so the docs should say "at least" rather than "more than", and the unit test backdates the advertisement to exactly the period. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses a Copilot review comment: track when a transaction was last advertised to each peer with Instant instead of SystemTime, so that wall-clock jumps (NTP corrections, VM clock adjustments) cannot perturb the re-advertisement schedule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses a Copilot review comment: `Instant - Duration` panics if the result would precede the monotonic clock's epoch (e.g. system boot). Use `checked_sub` and skip the test in that (practically unreachable) case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
rs/bitcoin/adapter/src/transaction_store.rs:348
- The test doc comment has duplicate step numbering ("3." appears twice). This makes the steps harder to follow when reading failures.
/// This function tests that we don't readvertise transactions that were recently advertised.
/// Test Steps:
/// 1. Add transaction to manager.
/// 2. Advertise that transaction and create requests from peer.
/// 3. Check that this transaction does not get advertised again during manager tick.
/// 3. Check transaction advertisement is correctly tracked.
|
✅ No security or compliance issues detected. Reviewed everything up to 61825bc. Security Overview
Detected Code Changes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
//rs/bitcoin/adapter:adapter_integration_testis one of the most flaky tests. All 4 flaky runs in the last week failed identically, in the single sub-testdoge_test_send_tx:i.e. the 1 DOGE sent from Alice to Bob never showed up within the test's 30-second polling window. Since Alice and Bob are labelled addresses in the same dogecoind wallet, the transaction is "trusted" and counts towards Bob's balance as soon as it enters dogecoind's mempool — so the failure means the transaction never reached dogecoind's mempool at all.
Root cause: the adapter's transaction relay is a one-shot with no retry
The relay path is:
TransactionStore::advertise_txidsadvertises a cached transaction to each peer exactly once: the peer's socket address is recorded inTransactionInfo.advertisedand never advertised to again until the transaction's 10-minute TTL expires. Thatadvertisedmark:send_toswallows connection write errors), andConnectionManager.The peer (dogecoind) only requests the transaction (
getdata) in reaction to theinv. So if any single step of the unacknowledgedinv → getdata → txhandshake is lost — e.g. theinvis written to a connection that is being torn down (5s incomplete-handshake timeout, ping timeout, invalid-message discard), or thegetdataarrives after the connection is gone — nobody ever retries and the transaction is silently lost for the rest of the test.The test binary runs 18 tests concurrently, each spawning bitcoind/dogecoind daemons, so CI runs see heavy CPU contention that provokes exactly this kind of connection churn and event-loop starvation. The DOGE variant flakes first because
dogecoindv1.14.9 is based on the much older Bitcoin Core 0.14 P2P stack, which is slower under load; the BTC variant runs identical test/adapter code and has not been observed failing.Prior deflake attempts (#9629, #8929, #8559, #7892, #7338, #7105) addressed other root causes (idleness, handshake waits, timeouts) but left the one-shot relay in place, which is why this test keeps recurring.
What
Make the transaction relay robust instead of a silent one-shot:
TransactionInfo.advertisednow records when the transaction was last advertised to each peer (HashMap<SocketAddr, Instant>instead ofHashSet<SocketAddr>), andadvertise_txidsre-advertises the transaction afterTX_READVERTISE_PERIOD_SECS(10s) for as long as it stays in the cache. The timestamps useInstant(a monotonic clock) rather thanSystemTime, so wall-clock jumps (NTP corrections, VM clock adjustments) cannot perturb the re-advertisement schedule.This retries the relay regardless of which message of the
inv → getdata → txexchange was lost — including across connection discard + reconnect. It is also safe on mainnet: peers that already received the transaction simply ignore the repeatedinvannouncement (a few dozen bytes per peer every 10s per still-cached transaction, worst case).While touching
advertise_txids, also fixed a latent bug in theMAXIMUM_TRANSACTION_PER_INVoverflow branch, where a shadowed loop variable broadcast the partially-built inventory of one peer to all peers (currently unreachable: the cache holds at most 250 transactions, far below the 50,000 threshold).Unit tests: added
test_adapter_readvertise_after_period, and adjusted the existingtest_adapter_dont_readvertise*tests, which still verify that transactions are not re-advertised within the re-advertisement period.Verification
bazel test --test_output=errors --runs_per_test=3 --jobs=3 //rs/bitcoin/adapter:adapter_integration_test— 3 parallel runs (reproducing the CI contention) all pass.bazel test //rs/bitcoin/adapter:adapter_test //rs/bitcoin/adapter:adapter_integration_test— pass.This PR was created following the steps in
.claude/skills/fix-flaky-tests/SKILL.md.🤖 Generated with Claude Code