High Metrics Cardinality Issue: tidb_tikvclient_rpc_err_total has error message in label

[hcc429]

Hi! We're seeing extremely high Prometheus cardinality for tidb_tikvclient_rpc_err_total because the type label contains the raw error message, and the error message sometime includes client/server addresses with ip:port or estimated wait time. This makes the label value effectively unbounded and creates a new time series for many occurrences.

Example

• {type="[PD:client:ErrClientResourceGroupThrottled]exceeded resource group quota limitation, estimated wait time 1m20.049602405s, ltb state is 1.20:-96.06"}
• {type="rpc error: code = Unavailable desc = error reading from server: read tcp 10.242.13.197:33174->10.242.11.10:20160: read: connection timed out"}

Current Implementation

internal/locate/region_request.go

	if errStr := getErrMsg(err); len(errStr) > 0 {
		metrics.TiKVRPCErrorCounter.WithLabelValues(getErrMsg(err), storeLabel).Inc()
	} else {
		metrics.TiKVRPCErrorCounter.WithLabelValues("unknown", storeLabel).Inc()
	}

Screenshots

Proposed fix

Instead of using the raw error message as the type label value, categorizing errors into a bounded set of known classes, e.g. type="throttled" for resource-group throttling and type="unknown" for everything else. This keeps label cardinality bounded and avoids embedding dynamic data (ip:port, durations, etc.) in labels.
I’m happy to contribute a PR once we agree on the direction

[ti-chi-bot]

Welcome @hcc429! It looks like this is your first issue to tikv/client-go 🎉