Document that Micrometer's "error" tag should be preferred vs. legacy "exception" tag

[chicobento]

On Spring Boot 3 besides exception, method, outcome, status and uri, the error tag is also being generated:

http_server_requests_count{error="none" ... }

This tag is not on official docs neither ServerHttpObservationDocumentation class and overlaps with exception tag.
Looking at the code it seems that this behavior is added by micrometer's DefaultMeterObservationHandler which is configured by ObservationAutoConfiguration.

A possible quick-fix would be to override DefaultMeterObservationHandler#createErrorTags to return Tags.empty().

On a side note, an additional undocumented behavior added by DefaultMeterObservationHandler is the http.server.requests.active metrics. Should this be documented ?

[chicobento]

Thread on micrometer slack on this issue: https://micrometer-metrics.slack.com/archives/C030GTHE4P6/p1672670045511959

[bclozel]

I agree with @jonatan-ivanov 's assessment in this Slack thread.

I see 3 solutions for this:

1. Removing the "extra" exception tag contributied by Spring MVC. This would break a lot of existing applications and dashboards.
2. Removing the error tag contributed by Micrometer itself, since we're using the Micrometer Observation API. This doesn't make any sense as Micrometer can be used without Spring and should enforce such common tags
3. Leave things as is and live with the fact that we have duplicate tags.

I think 3) is the best solution. We could revisit this in the future, but I can't think of a proper milestone for this change. Typically, metrics dashboards deal with multiple applications, built with various Spring Boot versions. Unless there's a strong driver for this removal, I don't think this is worth breaking people.

I'm going to turn this into a documentation improvement and call out in our reference documentation that the error tag should be preferred in dashboards and that exception might be removed in the future.

[chicobento]

@bclozel fair enough, sounds like a plan. just a heads up that http.server.requests.active metric will have the exception tag but not the error tag - not that either one or the other are relevant for this metric.
not sure about the behavior of other metrics though, will add here if I find anything.

[jonatan-ivanov]

@bclozel 👍🏼 Maybe it makes sense to provide the way in the docs to remove the exception tag if users want to:

@Bean
MeterFilter exceptionTagRemover() {
    return MeterFilter.ignoreTags("exception");
} 

A step forward could be (if users asks for it?) something similar that we have with common tags: if management.metrics.tags present, Boot creates a MeterFilter that adds those tags to every meter. We could do something very similar to ignore tags.

[jonatan-ivanov]

@chicobento As you mentioned neither exception nor error are relevant for http.server.requests.active. You probably already figured it out but these .active time series will never have error tag and the exception tag should always be none. The reason behind this is that they track operations that are in-progress (LongTaskTimer), once the operation failed, it is already terminated. So .active looks in the present and there you don't know if an operation failed or not since it is still in progress. The "normal" (w/o .active) time series look into the past, where you have the data if an operation failed or not.

Please let us know if you found any issues.