Skip to content

resource_control: delete metric children by creation-time keyspace_name#10910

Open
mvanhorn wants to merge 1 commit into
tikv:masterfrom
mvanhorn:fix/10871-keyspace-metric-leak
Open

resource_control: delete metric children by creation-time keyspace_name#10910
mvanhorn wants to merge 1 commit into
tikv:masterfrom
mvanhorn:fix/10871-keyspace-metric-leak

Conversation

@mvanhorn

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: Closes #10871

Resource Control metric caches in the resource manager are keyed by keyspace ID, resource group and RU type, but the underlying Prometheus children are created and removed by their label values, which include keyspace_name. When a metric child is first created while the keyspace-name cache misses, it carries a fallback label such as keyspace-123. Later, the periodic cleanup loop resolves the real keyspace name (for example tenant-a) from storage and calls DeleteLabelValues with that real name. Because creation used keyspace-123 but deletion uses tenant-a, the original child is never removed and leaks as a stale Prometheus series, leaving duplicated curves in Grafana for the same keyspace.

What is changed and how does it work?

The cached counter, gauge and max-per-sec tracker entries now remember the keyspace_name label string actually used when each child was created, and cleanup deletes by that recorded value rather than a freshly resolved one. This keeps the create and delete label values symmetric so no orphaned child remains:

  • counterMetrics and gaugeMetrics store the creation-time keyspaceName; deleteMetrics uses the cached value (falling back to the passed-in name only when no cache entry exists).
  • deleteMaxPerSecTracker removes the max-per-sec gauge children using the tracker's stored keyspaceName; those two deletions move out of the shared deleteLabelValues helper so they always use the creation-time value.
  • gaugeMetrics.setGroup deletes override-settings children by the struct's stored keyspaceName.

This mirrors the principle already applied to the request metrics in #10867.

resource_control: delete metric children by creation-time keyspace_name

Check List

Tests

  • Unit test

Code changes

  • No configuration, API, or persistent data change

Side effects

  • None

Release note

Fix a Resource Control metrics leak where Prometheus series created with a fallback keyspace_name label were not removed during cleanup.

The Resource Control metric caches are keyed by keyspace ID, group and RU
type, but Prometheus children are created and removed by their label values
including keyspace_name. When a child is created while the keyspace-name
cache misses, the fallback value keyspace-<id> is used. A later cleanup that
resolves the real keyspace name then calls DeleteLabelValues with the real
name, leaving the original keyspace-<id> child behind and leaking a stale
series.

Record the keyspace_name used when each cached counter, gauge and
max-per-sec tracker is first created, and delete using that stored value so
creation and deletion stay symmetric. This mirrors the approach taken in
PR tikv#10867 for the request metrics.

Fixes tikv#10871

Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 17, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Hi @mvanhorn. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Jun 17, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Welcome @mvanhorn!

It looks like this is your first PR to tikv/pd 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to tikv/pd. 😃

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 17, 2026
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.20%. Comparing base (5eb3dab) to head (7f9be67).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10910      +/-   ##
==========================================
+ Coverage   79.06%   79.20%   +0.14%     
==========================================
  Files         540      540              
  Lines       74535    74672     +137     
==========================================
+ Hits        58929    59142     +213     
+ Misses      11423    11356      -67     
+ Partials     4183     4174       -9     
Flag Coverage Δ
unittests 79.20% <91.66%> (+0.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rleungx rleungx left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix CI, the rest LGTM.

@ti-chi-bot

ti-chi-bot Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 18, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-06-18 08:18:33.202550501 +0000 UTC m=+1639214.272867891: ☑️ agreed by rleungx.

@ti-chi-bot ti-chi-bot Bot added the approved label Jun 18, 2026
@rleungx

rleungx commented Jun 18, 2026

Copy link
Copy Markdown
Member

/ok-to-test

@ti-chi-bot ti-chi-bot Bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-1-more-lgtm Indicates a PR needs 1 more LGTM. ok-to-test Indicates a PR is ready to be tested. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resource_control: fallback keyspace_name labels can leak in metrics cleanup

2 participants