Skip to content

Pipeline should have middleware and be configured #1513

@delner

Description

@delner

Currently, the Datadog::Pipeline is a global-level class that has mutable state. It is configured via Datadog::Pipeline.before_flush { } and used via Datadog::Pipeline.process!.

Some downsides:

  • It is not possible to "reset" the state of the pipeline (once its modified, its permanent until application restart.)
  • Its state is effectively a global variable, which has negative consequences (particularly Ractors)

It would be better if:

  1. Pipeline was instantiated, rather than accessed through class methods. Perhaps middleware style?
    Datadog::Pipeline.new do |p|
      # Custom processor: passed "traces", returns traces
      p.process HealthCheckTraceFilter
      # Shorthand for processing traces in aggregate
      # Return traces to be kept
      p.process { |traces| }
      # Shorthand for filtering each individual trace
      # Return truthy to keep, falsey to remove
      p.trace_filter { |trace| }
      # Shorthand for filtering each individual span
      # Return truthy to keep, falsey to remove
      p.span_filter { |span| }
    end
  2. Pipeline was managed by configuration e.g.
    Datadog.configure do |c|
      c.pipeline do |p|
        p.process HealthCheckTraceFilter
        # More pipeline middleware here...
      end
    end

This would make it easier to rebuild and modify the pipeline in a controlled, non-ambiguous way.

Metadata

Metadata

Assignees

Labels

coreInvolves Datadog core librariesdev/refactorInvolves refactoring existing componentsfeature-requestA request for a new feature or change to an existing one

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions