Skip to content

test: stabilize flaky TestHotTestSuite/TestBuckets#10937

Open
flaky-claw wants to merge 1 commit into
tikv:masterfrom
flaky-claw:flakyfixer/case_cf396bbf1769-a6
Open

test: stabilize flaky TestHotTestSuite/TestBuckets#10937
flaky-claw wants to merge 1 commit into
tikv:masterfrom
flaky-claw:flakyfixer/case_cf396bbf1769-a6

Conversation

@flaky-claw

@flaky-claw flaky-claw commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #10651

Problem Summary:
Flaky test TestHotTestSuite/TestBuckets in github.com/tikv/pd/tools/pd-ctl/tests/hot intermittently fails, so this PR stabilizes that path.

What changed and how does it work?

Root Cause

TestBuckets teardown could call CleanCache while a late async write hot-cache task still touched HotPeerCache maps.

Fix

The final write drain serializes teardown behind the last write-side hot-cache task without changing CLI assertions or product behavior.

Verification

Spec:

  • target: github.com/tikv/pd/tools/pd-ctl/tests/hot :: TestHotTestSuite/TestBuckets
  • strategy: pd.issue_scoped.v1
  • plan mode: BASELINE_ONLY
  • requirements: required case must execute; no skip; repeat count = 1
  • baseline gates: required_flaky_gate, build_safety_gate, intent_guard_gate
  • feedback surface source: baseline_only

Observed result:

  • status: failed
  • required case executed: yes
  • submission decision: ALLOWED
  • note: Required flaky case executed during validation.
    Required flaky case was not skipped.
    nested_target_json passed.
    race_target passed.

Gate checklist:

  • raw_target_path_mismatch: SKIPPED
  • raw_package_path_mismatch: SKIPPED
  • nested_target_json: PASS
  • race_target: PASS
  • build: BLOCKED

Commands:

  • cd tools && go test -json -tags without_dashboard ./pd-ctl/tests/hot -run '^TestHotTestSuite/TestBuckets$' -count=1
  • cd tools && go test -race -tags without_dashboard ./pd-ctl/tests/hot -run '^TestHotTestSuite/TestBuckets$' -count=5

Release note

None

Fixes #10651

Summary by CodeRabbit

  • Tests
    • Added an extra check to ensure hot peer statistics are populated correctly after asynchronous updates, improving confidence in hot-spot handling behavior.

@ti-chi-bot ti-chi-bot Bot added the release-note-none Denotes a PR that doesn't merit a release note. label Jun 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hundundm for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added contribution This PR is from a community contributor. dco-signoff: no Indicates the PR's author has not signed dco. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Hi @flaky-claw. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot

ti-chi-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.

📝 Please follow instructions in the contributing guide to update your commits with the DCO

Full details of the Developer Certificate of Origin can be found at developercertificate.org.

The list of commits missing DCO signoff:

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ti-chi-bot ti-chi-bot Bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

TestBuckets now asserts that write hot peer stats exist after async hot peer updates are drained, before test teardown clears the hot cache.

Changes

Hot peer cache test

Layer / File(s) Summary
Bucket stats assertion
tools/pd-ctl/tests/hot/hot_test.go
Adds a non-nil check for hotStat.GetHotPeerStats(utils.Write, 0) after draining async hot peer updates.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Possibly related PRs

  • tikv/pd#10863: Also updates checkBuckets in the same test area to assert hot peer stats are populated for bucket index 0 after async updates.

Suggested labels

size/XS, ok-to-test, contribution, dco-signoff: yes

Suggested reviewers

  • lhy1024
  • rleungx

Poem

🐰 I hopped through buckets, soft and keen,
A write hot stat now must be seen.
One tiny check, before reset's bite,
Keeps the cache calm through day and night.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states the main change: stabilizing the flaky TestHotTestSuite/TestBuckets test.
Description check ✅ Passed The PR description covers the problem, linked issue, fix summary, verification, and release note; only checklist detail is light.
Linked Issues check ✅ Passed The fix serializes teardown behind the last write-side hot-cache task, directly addressing the HotPeerCache race in TestBuckets.
Out of Scope Changes check ✅ Passed The change stays confined to the flaky test path and adds a targeted assertion, with no unrelated behavior changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/pd-ctl/tests/hot/hot_test.go`:
- Line 558: The drained write stats check in hot_test.go is vacuous because
hotStat.GetHotPeerStats(utils.Write, 0) returns an allocated map even when
empty, so reusing re.NotNil does not verify population. Update the assertion
near the existing hotStat.GetHotPeerStats(utils.Write, 0) call in the hot test
to check for a non-empty result instead, using a size/length-based assertion or
equivalent on the returned map so the test actually confirms drained write peers
were recorded.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e871b664-3b07-479c-bd56-d6d5a427c97a

📥 Commits

Reviewing files that changed from the base of the PR and between 51f4146 and 94277ce.

📒 Files selected for processing (1)
  • tools/pd-ctl/tests/hot/hot_test.go

// Drain async hot peer updates before TearDownTest resets the hot cache directly.
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use a non-vacuous assertion for drained write stats

Line 558 duplicates Line 556, and NotNil is ineffective here because GetHotPeerStats returns an allocated map even when no hot peers exist. This does not validate the drain/population condition.

Suggested fix
-	re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
-	re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
-	re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
+	writeStats := hotStat.GetHotPeerStats(utils.Write, 0)
+	re.NotEmpty(writeStats)
+	found := false
+	for _, peers := range writeStats {
+		if len(peers) > 0 {
+			found = true
+			break
+		}
+	}
+	re.True(found)
+	re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
writeStats := hotStat.GetHotPeerStats(utils.Write, 0)
re.NotEmpty(writeStats)
found := false
for _, peers := range writeStats {
if len(peers) > 0 {
found = true
break
}
}
re.True(found)
re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/pd-ctl/tests/hot/hot_test.go` at line 558, The drained write stats
check in hot_test.go is vacuous because hotStat.GetHotPeerStats(utils.Write, 0)
returns an allocated map even when empty, so reusing re.NotNil does not verify
population. Update the assertion near the existing
hotStat.GetHotPeerStats(utils.Write, 0) call in the hot test to check for a
non-empty result instead, using a size/length-based assertion or equivalent on
the returned map so the test actually confirms drained write peers were
recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. dco-signoff: no Indicates the PR's author has not signed dco. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestBuckets has a HotPeerCache data race

1 participant