Mutable global labels support?

[pcholakov]

Hey folks,

We use metrics-exporter-prometheus in network server, and we would like to apply a global label that isn't immediately known on startup. It's useful to us to begin recording metrics immediately, and only add the label once it is known, which could take a second as it requires communicating with a remote service.

In light of this, would you consider making the global labels mutable after the recorder is initialized? I sketched out how this might work here: 7c820d8. I had to carefully splice the new labels into the distributions structure; it worked in my testing but admittedly it wasn't very extensive. Would you consider such a feature? I can open a PR and we can continue the conversation there, if so.

In any event, thanks for an incredibly useful crate!

(Here is the context in which this requirement came up: restatedev/restate#3027)

[tobz]

My gut reaction is that I dislike it: it means that all metrics have the potential to disappear, and then reappear, out of thin air, with new labels. Potential issue with counter reset behavior kicking in.. but that said, if someone wants to do this, that's on them, I suppose.

That said, I'm willing to review a PR adding this behavior, and review it in good faith that I'm willing to merge such a thing.

I have to remind myself how we apply global labels -- during ingestion or during rendering -- but I think that would be the main problem in my mind: avoiding complexity with how the labels manage to affect metrics, whether existing or new.

[pcholakov]

Agreed, that's exactly the concern that I have, so I was hesitant to even bring this up.

Supposing this is implemented correctly, the issue (for those who choose to use this feature) is that when scraping metrics before and after global labels have been modified, the caller will get a whole new view of the metrics. I took care not to reset any counters in the process but they essentially disappear and reappear with new labels in the process.

If I understand the distinction correctly, global labels are currently applied during ingestion. The distributions map is laid out based on the supplied labels, and this directly influences how the metrics will be rendered later.

A more composable approach to achieve the end goal could be to add support for a user-supplied filter function to be applied during rendering. As is, the only way to patch the global labels is to parse the String that PrometheusHandle::render returns. If we could stream the metric lines out, it would be a lot easier to post-process the output and not require any brittle patching of the internal aggregations.