OTEP: Context-scoped Attributes#4931
OTEP: Context-scoped Attributes#4931carlosalberto wants to merge 7 commits intoopen-telemetry:mainfrom
Conversation
| (i.e., it is not necessary to make Context-scoped attributes distinguishable | ||
| from “direct” telemetry attributes). | ||
|
|
||
| ### Exporter changes (none required) |
There was a problem hiding this comment.
its best to just remove this section completely.
| } | ||
| ``` | ||
|
|
||
| The Context-scoped attributes must be available for all telemetry items they |
There was a problem hiding this comment.
we need to specify something about how it gets added to log/span/metrics. This OTEP says opt-in, but does not say anything specific beyond that. Is that intentionally left out in OTEP
There was a problem hiding this comment.
Clarified this to (very) succinctly mention that the Context-scoped attributes are added at telemetry creation/emit time.
| # Context-scoped attributes | ||
|
|
||
| Add Context-scoped telemetry attributes which typically apply to all signals | ||
| associated with a trace as it crosses a single service. |
There was a problem hiding this comment.
do we need to mention "trace" here?
There was a problem hiding this comment.
Updated to "request" to make it more general.
| ## Motivation | ||
|
|
||
| This OTEP aims to address various related demands that have been brought up | ||
| in the past, where the scope of resource attributes is too broad, but the scope |
There was a problem hiding this comment.
not just the scope, but the lifetime too. Or it is implied I guess.
| This OTEP aims to address various related demands that have been brought up | ||
| in the past, where the scope of resource attributes is too broad, but the scope | ||
| of span attributes is too narrow. For example, this happens where there is a | ||
| mismatch between the OpenTelemetry SDK’s (and thus TracerProvider’s, MeterProvider’s) |
| between these, other than with the generic HTTP attributes like | ||
| `http.route` or `url.template`. The mentioned issue proposes to add an | ||
| `http.app` attribute but it is unclear where this could be placed. | ||
| The resource cannot be used, since there is only one that is shared between |
There was a problem hiding this comment.
resource cannot be used - this seems like a very specific limitation for the jvm example? Isn't it possible to create multiple Meter/Tracer/Logger providers, each with its own resource, for each independent application within the same process?
There was a problem hiding this comment.
Included this as an alternative mentioned in the Motivation section (albeit being a prohibitive one, given you would need to duplicate the pipeline).
| booleans or arrays thereof, just like for span or resource attributes. | ||
| Context-scoped attributes are associated with all telemetry signals emitted while | ||
| the Context containing the Context-scoped attributes is active and are available | ||
| to samplers, processors and exporters. For spans, the context within which the span |
There was a problem hiding this comment.
exporters? That would be incorrect as a batching_processor can call exporter in its background thread, losing the original context.
There was a problem hiding this comment.
Clarified this. Context-scoped attributes only apply to telemetry at creation time.
| to samplers, processors and exporters. For spans, the context within which the span | ||
| is started applies. Like other telemetry APIs, Context-scoped attributes are | ||
| write-only for applications. You cannot query the currently set Context-scoped attributes, | ||
| as they are only available on the SDK level (to samplers, processors and exporters). |
There was a problem hiding this comment.
Removed them.
| must be implemented for them. This is because they are meant to annotate (a subset of) | ||
| the spans of a single service (see also [the next section](#comp-baggage)). | ||
|
|
||
| Context-scoped attributes MUST be disabled by default. They MUST be offered as an opt-in |
There was a problem hiding this comment.
lets make it clear what is meant by disabled here. I guess you are referring to "Context-scoped attributes are not automatically added to spans/logs/metrics unless user has explicitly opted into it. Opt-in mechanism are in [link to the section where we talk about it.]"
There was a problem hiding this comment.
I didn't see any text describing what an opt-in mechanism would look like. I expected it to take the form of config params on SDK TracerProvider, LoggerProvider, MeterProvider, or new built-in SpanProcessor, LogRecordProcessors, MeasurementProcessor (rejected 🙁) with corresponding config handles.
Did I miss something or is the opt-in mechanism TBD?
|
|
||
| However, the use cases are very different: Baggage is meant to transport app-level | ||
| data along the path of execution, while Context-scoped attributes are annotations | ||
| for telemetry in the same execution path. |
There was a problem hiding this comment.
app-level data vs telemetry annotations - not sure of the differences between them.. In my view, the only difference between Baggage is that Baggage is out-of-proc+in-proc, whereas Context-scope Attributes is in-proc only.
(I can think of calling this as InProcOnlyBagagge as well)
What kind of data users put inside it - totally upto them. They can put data to affect control flow, telemetry attributes, feature-flags etc. OTel don't put any restrictions.
There was a problem hiding this comment.
Baggage only accepts strings. Also, if we were to go any Baggage way, we should offer automatic attachment of Baggage values at the SDK level, like the one defined by this OTEP.
| Context-scoped attributes would be implementable, it would break the intended | ||
| meaning by extending the scope of attributes to called services and would also | ||
| raise many security concerns. | ||
| * Baggage is both readable and writable for the application, while Context-scoped |
There was a problem hiding this comment.
I am not so sure about "write-only" aspect. Why restrict it, when Context itself supports reading:
specification/tree/main/specification/context#get-value
There was a problem hiding this comment.
I'd like to reduce the area if possible. If we get cases were it is valuable to expose this to the public, we can go ahead and do that.
| * Baggage is both readable and writable for the application, while Context-scoped | ||
| attributes, like all other telemetry attributes, are write-only. | ||
| * Baggage only supports string values. | ||
| * Baggage is not available to telemetry exporters (although e.g., a SpanProcessor |
There was a problem hiding this comment.
Left this same comment elsewhere too, but repeating here as this statement feels incorrect to me. Exporters can always access a Context, but that is very unlikely the one present during the actual telemetry creation.
There was a problem hiding this comment.
Good call, made a clarification.
| * Both apply a set of telemetry attributes to a set of telemetry items | ||
| (those in their "scope"). | ||
|
|
||
| While Instrumentation Scope attributes apply to all items emitted by the same |
There was a problem hiding this comment.
I think a good way to describe this is - they differ in lifetime. Tracer/Meter etc. typically lives same time as the application itself. But context-scoped ties its lifetime to the context itself, so if the context is per-request, then it lives only that long. But if the context is app-start, then it lives entire application.
There was a problem hiding this comment.
Added a small paragraph to mention this.
| * If the context already contains any attributes with that name, they MUST be | ||
| overwritten with the attributes of the later call. | ||
| * If an associated telemetry item (e.g. Span) already has an attribute with | ||
| a name that is also in the Context-scoped attributes, the telemetry attribute |
There was a problem hiding this comment.
this API section is merely talking about adding ScopedAttribtues to Context - so it is better to avoid the mentioning of Span case here (and span attributes)
There was a problem hiding this comment.
Actually I think examples are useful in order to be super clear. Changed this to "e.g. Span or LogRecordItem".
|
|
||
| The Context-scoped attributes must be available for all telemetry items they | ||
| are logically associated with. For example, the `ReadableSpan` should include | ||
| the context attributes in the list of Span attributes, and they must be included |
There was a problem hiding this comment.
This section feels a bit odd - we directly jump into the state where Span attributes already contain Context attributes, without specifying who/how that happened.
There was a problem hiding this comment.
Removed this paragraph and simplified the previous one - it may be too short but it's now more explicit.
|
|
||
| This OTEP relies on the | ||
| [`Context`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/context/README.md) | ||
| concept that has been in the spec since 1.0. However, at least .NET went 1.0 |
There was a problem hiding this comment.
Don't think this is true - .NET also has context. Used by Baggage.
(Its span (activity)/span context are implemented outside of this context. It can be an issue in some cases. But those cases already have issues with Baggage too.).
Its better to avoid including .NET's exceptions to the OTEP.
There was a problem hiding this comment.
.NET folks anyway has to deal with a competing concept in ILogger - it has ILogger.BeginScope()., quite similar to this proposal - but sadly, it's tied to loggging signal only and not general purpose like this proposal.
| is injected in a JVM running a classical Java Application Server (such as tomcat), | ||
| there will be a single resource for the whole JVM. The application server can | ||
| host multiple independent applications. There is currently no way to distinguish | ||
| between these, other than with the generic HTTP attributes like |
There was a problem hiding this comment.
I think this is the wrong example because we have another OTEP #4665 which targets this same use case.
The classic example of the value of this feature for me is a service serving multi tenant traffic:
- Every request is associated with a tenant, which is resolved from something like a header, access token, standard request param, etc
- The same service handles requests for many many tenants. Its not like the application server example with one application per tenant, with maybe tens of distinct applications. It could be thousands, tens of thousands, or more running through a single application.
- Its critical that some or all telemetry be associated with this telemetry. I.e. metrics, logs, and traces, and from a variety of different scopes recording instrumentation.
With something like the java agent, its not possible to modify all the instrumentation needed to capture this tenant information. You could put it into a custom context key, and write a custom log record processor, custom span processor to extract it from context and stamp it onto spans and logs, but what about metrics? Also, since this is a common requirement, why don't we make it easy instead of requiring users to build a bunch of custom machinery to accomplish it?
^^ Just giving ideas about an alternative example you could use.
There was a problem hiding this comment.
Thanks for the suggestions. Re-massaged the section, including a pair of new paragraphs with this information.
| is executed in is lost in any child spans that may occur within the function (app). | ||
|
|
||
| There is also the issue | ||
| [open-telemetry/opentelemetry-specification#1089](https://github.com/open-telemetry/opentelemetry-specification/issues/1089) |
There was a problem hiding this comment.
FYI, this issue is solved. Currently, my read is that the text implies that this is an open problem. Despite being solved, its probably still relevant and maybe a part of the solution. I.e. perhaps we define built in processors to make it easy to life context scoped attributes onto telemetry. The span processor would be able to leverage BeforeEnd.
There was a problem hiding this comment.
Oh, good point. I moved to the prior art section (IMHO it could be useful for people but not as crucial as before).
| signals emitted within a Context. Context-scoped attributes are standard | ||
| attributes, which means you can use strings, integers, floating point numbers, | ||
| booleans or arrays thereof, just like for any telemetry item. | ||
| Context-scoped attributes MUST be added to telemetry items that were **emitted** |
There was a problem hiding this comment.
this MUST here somewhat conflicts with the "MUST be disabled by default" later.
3 participants
Add Context-scoped telemetry attributes which typically apply to all signals associated with a trace as it crosses a single service.
Resurrecting open-telemetry/oteps#207 while updating some changes that have happened since (e.g. changes in
InstrumentationScopebeing a runtime concept rather than a compile-time one). Originally proposed by @Oberon00Additionally:
Processoralternative being discarded as we would need to add an additional preOnStartcall in order to support Samplers.Baggageshould be included as part of stamping attributes to individual telemetry items (similar enough functionality).Java prototype for traces/logs: https://github.com/open-telemetry/opentelemetry-java/compare/main...carlosalberto:context-scoped-attrs-proto?expand=1
Summary
This OTEP aims to address various related demands that have been brought up in the past, where the scope of Resource attributes is too broad, but the scope of Span attributes is too narrow. For example, this happens where there is a mismatch between the OpenTelemetry SDK’s (and thus TracerProvider’s, MeterProvider’s) process-wide initialization and the semantic scope of a (sub)service.
The context-scoped attributes allows you to attach attributes to all telemetry signals emitted within a Context. Context-scoped attributes are normal attributes, which means you can use strings, integers, floating point numbers, booleans or arrays thereof, just like for Span or Resource attributes. Context-scoped attributes are associated with all telemetry signals emitted while the Context containing the Context-scoped attributes is active and are available to telemetry exporters. For spans, the context within which the span is started applies. Like other telemetry APIs, Context-scoped attributes are write-only for applications. You cannot query the currently set Context-scoped attributes, they are only available on the SDK level (e.g. to Samplers and SpanProcessors and exporters).
Context-scoped attributes should be thought of equivalent to adding the attribute directly to each single telemetry item it applies to.