resource_control: delete metric children by creation-time keyspace_name#10910
resource_control: delete metric children by creation-time keyspace_name#10910mvanhorn wants to merge 1 commit into
Conversation
The Resource Control metric caches are keyed by keyspace ID, group and RU type, but Prometheus children are created and removed by their label values including keyspace_name. When a child is created while the keyspace-name cache misses, the fallback value keyspace-<id> is used. A later cleanup that resolves the real keyspace name then calls DeleteLabelValues with the real name, leaving the original keyspace-<id> child behind and leaking a stale series. Record the keyspace_name used when each cached counter, gauge and max-per-sec tracker is first created, and delete using that stored value so creation and deletion stay symmetric. This mirrors the approach taken in PR tikv#10867 for the request metrics. Fixes tikv#10871 Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
Hi @mvanhorn. Thanks for your PR. I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Welcome @mvanhorn! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #10910 +/- ##
==========================================
+ Coverage 79.06% 79.20% +0.14%
==========================================
Files 540 540
Lines 74535 74672 +137
==========================================
+ Hits 58929 59142 +213
+ Misses 11423 11356 -67
+ Partials 4183 4174 -9
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
rleungx
left a comment
There was a problem hiding this comment.
Please fix CI, the rest LGTM.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rleungx The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
/ok-to-test |
What problem does this PR solve?
Issue Number: Closes #10871
Resource Control metric caches in the resource manager are keyed by keyspace ID, resource group and RU type, but the underlying Prometheus children are created and removed by their label values, which include
keyspace_name. When a metric child is first created while the keyspace-name cache misses, it carries a fallback label such askeyspace-123. Later, the periodic cleanup loop resolves the real keyspace name (for exampletenant-a) from storage and callsDeleteLabelValueswith that real name. Because creation usedkeyspace-123but deletion usestenant-a, the original child is never removed and leaks as a stale Prometheus series, leaving duplicated curves in Grafana for the same keyspace.What is changed and how does it work?
The cached counter, gauge and max-per-sec tracker entries now remember the
keyspace_namelabel string actually used when each child was created, and cleanup deletes by that recorded value rather than a freshly resolved one. This keeps the create and delete label values symmetric so no orphaned child remains:counterMetricsandgaugeMetricsstore the creation-timekeyspaceName;deleteMetricsuses the cached value (falling back to the passed-in name only when no cache entry exists).deleteMaxPerSecTrackerremoves the max-per-sec gauge children using the tracker's storedkeyspaceName; those two deletions move out of the shareddeleteLabelValueshelper so they always use the creation-time value.gaugeMetrics.setGroupdeletes override-settings children by the struct's storedkeyspaceName.This mirrors the principle already applied to the request metrics in #10867.
Check List
Tests
Code changes
Side effects
Release note