otel.sdk.log.created for filtered logs

[anuraaga]

I have been working on implementing SDK metrics. Currently, it is implemented to count only emitted logs in Java and I am working on the same implementation for other languages.

https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/logs/src/main/java/io/opentelemetry/sdk/logs/SdkLogRecordBuilder.java#L152

The question came up in JS on whether we should apply this to filtered logs or not

open-telemetry/opentelemetry-js#6433 (comment)

I suspect the answer for the current conventions that don't define attributes is no. But it seems reasonable that filtered logs are also counted somehow. A couple of approaches

• Define a new otel.sdk.log.filtered metric for filtered logs
• Define an attribute, otel.log.enabled to set to true or false for filtered or emitted logs.

The second proposal is more consistent with otel.sdk.span.started, however one can argue that spans are always started and ended, even when non-recording, while filtered logs are not actually created so shouldn't be in the counter.

[thompson-tomo]

Isn't the qty of filtered the difference between otel.sdk.log.created (incoming) and otel.sdk.exporter.log.exported (outgoing)?

There is some discussion on this topic in #3470 as some thoughts/proposals.

[anuraaga]

IIUC it may not be flexible enough to compare metrics like that, notably if more attributes are added to otel.sdk.log.created, there may not be an appropriate union between the metrics which are generally not related to each other.

Actually I realized that the instrumentation scope for these logs is also vague - I think the instrumentation scope of log records is usually the logger name (otherwise the extremely important logger name would be lost).

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/log4j/log4j-appender-2.17/library/src/main/java/io/opentelemetry/instrumentation/log4j/appender/v2_17/OpenTelemetryAppender.java#L314C12-L314C31

Currently I implemented otel.sdk.log.created in an opentelemetry-sdk type of scope, but maybe it should have been the same scope as the log record? Or otherwise there may need to be an attributed added to otel.sdk.log.created to be able to have counts per logger. Either way, it seems that otel.sdk.exporter.log.exported wouldn't behave the same way as it is an exporter shared among loggers so wouldn't be logger-scoped, and then we wouldn't be able to take a valid difference.

#3470 looks quite related but it seems to be for the processors/exporters, and about performance, rather than filtering at the Logger level based on Logger config. Maybe they should have a similar attribute though.

[thompson-tomo]

My impression is the *.created metrics are all light on attributes given the intention of capturing information early. I think they always be related given it is part of the pipeline. I think using the original scope is the intention based on attributes capturing the component. But this needs guidance given it is metrics.

it seems to be for the processors/exporters, and about performance, rather than filtering at the Logger level based on Logger config.

The key part is #3470 (comment) and wouldn't that filtering be done via a processor/exporter?

[anuraaga]

wouldn't that filtering be done via a processor/exporter?

Don't think so - the spec defines filtering by config at the Logger, not processor pipeline. I think this is because the intent is for the filtering to be configurable per-logger, while the pipeline is shared.

https://opentelemetry.io/docs/specs/otel/logs/sdk/#emit-a-logrecord

For example, Java implementation before the pipeline.

https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/logs/src/main/java/io/opentelemetry/sdk/logs/SdkLogRecordBuilder.java#L144

[thompson-tomo]

Hmm, I think from a user experience the processor metric still fits best for me as it considers that the filtering based on severity can occur anywhere. Note we could identify them by using a corresponding otel.component.type ie log_reciever and I consider the level filter as a builtin filter.

It might help to remove ambiguity if otel.sdk.log.created is renamed to otel.sdk.log.submitted given the description talks about logs submitted to the sdk.