Share Span/Trace ID with other eBPF based components by florianl · Pull Request #1139 · open-telemetry/opentelemetry-ebpf-profiler

florianl · 2026-01-29T10:46:42Z

Just wanted to share the PoC for open-telemetry/opentelemetry-specification#4855.

Not sure it should be merged at this point as the specification might still change.

Using open-telemetry/opentelemetry-ebpf-instrumentation#1184 and this PR will make sure that components share span/trace IDs - while the profiler is just a consumer and OBI produces these IDs.

A configuration option for BPF FS is used, as not all deployments use the same (default) location.

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

christos68k · 2026-02-03T09:22:15Z

  UnwindState *state = &record->state;

  maybe_add_apm_info(trace);
+  maybe_add_otel_span_trace_id(trace);


This should take into account maybe_add_apm_info. For example, if maybe_add_apm_info completes successfully then we should not be executing maybe_add_otel_span_trace_id as it could be redundant and will overwrite previous data.

We should also decide which source is the highest-fidelity one in order to resolve such conflicts: is it APM since its data comes from SDK instrumentation and is expected to fail less often?

Instead of adding complexity, I suggest moving forward with #1153.

maybe_add_apm_info implements a vendor specific option that is not adapted in the OTel community.

We can treat maybe_add_apm_info as a placeholder for the upcoming OTel thread context specification: the exact same conflict I described here will be present, so while we're still at the step of discussing/reviewing specifications it's good to take that into account in order to come up with a better solution.

Even if we don't plan to merge this PR now (as the OTEP isn't final yet), it's still useful to discuss.

if maybe_add_apm_info is kept, imho it should be higher fidelity as those info are set directly by the target process/sdk (so it should know better)

We still can continue with #1153 - its ideas remain in the git history and are not lost by removing apmint. It also leaves the room of discussion which data source is more accurate and should be preferred. From a OTel perspective, apmint is not relevant as it is vendor specific and just adds complexity - both in eBPF space but also in the user space counterpart.

christos68k · 2026-02-03T09:23:25Z

  UnwindState *state = &record->state;

  maybe_add_apm_info(trace);
+  maybe_add_otel_span_trace_id(trace);


Since this function is OBI-specific maybe we should rename it to reflect OBI for now. There will be multiple OTEL specifications for sharing trace/span ids.

OBI just happens to be the first to implement the specification open-telemetry/opentelemetry-specification#4855. Other OTel components, like OTel network, are also interested in this approach and plan to implement it.

As the implementation also is based on the OTel proposed specification and not OBI details, I suggest to not make it OBI specific.

But the specification is OBI-specific, it doesn't work without OBI as that's the source of the data. There is another OTel specification under discussion that is not OBI-specific. A function maybe_add_otel_span_trace_id that only refers to the first but not the latter is confusing so we should do a better job of disambiguating.

This specification is not OBI-specific, as this map also can be populated by OTel network. Profiling is just the beneficator of data that is provided by these other OTel components.

The required precedence of data source accuracy must be defined and documented once the discussion evolves into a specification, or even earlier, as an integral part of the ongoing discussion.

At the current state, I think, this should not block going forward with this approach.

I still want to see the fidelity/priority issue clarified before merging this PR so I'll wait until the specification is approved to revisit this point here.

@christos68k with 8458283 I updated the logic, to only use the OTel span/trace ID if span/trace ID are not yet set. Please let me know if this works for you.

Co-authored-by: Mattia Meleleo <melmat@tuta.io>

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

mmat11

lgtm!

christos68k

For context: @mmat11 reached out to me recently and mentioned that the currently open OTEP PR will be closed.

The specification for sharing this information will be moved to the OBI repository.

This means that we can move forward with the integration here and don't need to wait for the specification to be approved any more.

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

christos68k

Final batch of specification link fixes

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

rogercoll

LGTM (follow up: it would be great having a mechanism to optionally load ebpf maps)

Co-authored-by: Roger Coll <roger.coll@elastic.co>

florianl added 4 commits January 23, 2026 17:24

POC: use shared map for span/trace ID lookups

5bb137d

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Merge branch 'main' into share-span-trace-id

bbb170e

add configuration options for bpf fs

944701d

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Merge branch 'main' into share-span-trace-id

a088367

This comment was marked as outdated.

Sign in to view

florianl added 2 commits February 3, 2026 09:47

Merge branch 'main' into share-span-trace-id

2303e7e

update to latest spec changes

01cc2ff

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

florianl changed the title ~~PoC: Share Span/Trace ID with other eBPF based components~~ Share Span/Trace ID with other eBPF based components Feb 3, 2026

florianl marked this pull request as ready for review February 3, 2026 09:12

florianl requested review from a team as code owners February 3, 2026 09:12

christos68k reviewed Feb 3, 2026

View reviewed changes

mmat11 reviewed Feb 3, 2026

View reviewed changes

Comment thread tracer/tracer.go Outdated

Comment thread tracer/tracer.go Outdated

Comment thread tracer/tracer.go

Comment thread tracer/tracer.go Outdated

florianl and others added 5 commits February 3, 2026 11:31

Update tracer/tracer.go

136fbc0

Co-authored-by: Mattia Meleleo <melmat@tuta.io>

set map max_entries to 1 if BPF FS is not available

1289736

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

just log errors with shared map

62f643d

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

fix typo

2f97a80

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Merge branch 'main' into share-span-trace-id

b4e5c56

This was referenced Feb 5, 2026

remove apmint #1153

Open

OTEP: correlating OBI traces to profiles open-telemetry/opentelemetry-specification#4855

Closed

florianl and others added 10 commits February 9, 2026 16:00

Merge branch 'main' into share-span-trace-id

d233b8b

Merge branch 'main' into share-span-trace-id

d0ccced

Merge branch 'main' into share-span-trace-id

4c8824b

Merge branch 'main' into share-span-trace-id

36fb5fc

Merge branch 'main' into share-span-trace-id

34ad348

Merge branch 'main' into share-span-trace-id

9f1edd5

Merge branch 'main' into share-span-trace-id

9c24d9a

Merge branch 'main' into share-span-trace-id

6b990f1

Merge branch 'main' into share-span-trace-id

73d2b53

Merge branch 'main' into share-span-trace-id

beec6f2

only populate OTel span/trace ID if not yet set

8458283

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

mmat11 approved these changes Mar 30, 2026

View reviewed changes

florianl requested a review from christos68k March 30, 2026 12:29

christos68k reviewed Mar 30, 2026

View reviewed changes

Comment thread support/ebpf/interpreter_dispatcher.ebpf.c Outdated

Comment thread tracer/tracer.go Outdated

Comment thread cli_flags.go Outdated

florianl and others added 2 commits March 31, 2026 08:54

Apply suggestions from code review

c0b5ed3

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

update link

0e28ed3

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

christos68k approved these changes Mar 31, 2026

View reviewed changes

Comment thread support/ebpf/interpreter_dispatcher.ebpf.c Outdated

Comment thread support/ebpf/interpreter_dispatcher.ebpf.c Outdated

Comment thread support/ebpf/interpreter_dispatcher.ebpf.c Outdated

florianl and others added 6 commits April 2, 2026 09:37

Apply suggestions from code review

b6fff5e

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

Merge branch 'main' into share-span-trace-id

39ce509

connect bits

8c16687

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

fix suggested change

f6ff289

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Update support/ebpf/interpreter_dispatcher.ebpf.c

4c454ff

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

update bpf blobs

a3ee2f8

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

florianl requested a review from rogercoll April 2, 2026 08:39

Merge branch 'main' into share-span-trace-id

2a9ff3c

rogercoll reviewed Apr 7, 2026

View reviewed changes

Comment thread tracer/tracer.go Outdated

Comment thread tracer/tracer.go

Comment thread collector/factory.go

florianl and others added 4 commits April 7, 2026 14:22

use const for map name

f2d6224

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

introduce obi_process_ctx

c65b631

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

Merge branch 'main' into share-span-trace-id

77e0155

refactor code

dacfcd5

Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

rogercoll reviewed Apr 8, 2026

View reviewed changes

Comment thread tracer/tracer.go Outdated

Comment thread collector/factory.go

florianl and others added 2 commits April 8, 2026 07:52

Merge branch 'main' into share-span-trace-id

b1c82ea

Update tracer/tracer.go

a481247

Co-authored-by: Roger Coll <roger.coll@elastic.co>

rogercoll approved these changes Apr 8, 2026

View reviewed changes

florianl added the ready-to-merge label Apr 8, 2026

christos68k merged commit d20c77c into open-telemetry:main Apr 9, 2026
32 checks passed

Conversation

florianl commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

christos68k Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmat11 left a comment

Choose a reason for hiding this comment

Uh oh!

christos68k left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christos68k left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rogercoll left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florianl commented Jan 29, 2026 •

edited

Loading

christos68k Feb 3, 2026 •

edited

Loading

rogercoll left a comment •

edited

Loading