feat(debugger): add Remote Enablement support #7137

watson · 2025-12-18T07:53:22Z

What does this PR do?

Implements Remote Enablement (RE) for Dynamic Instrumentation/Live Debugger via APM_TRACING_MULTICONFIG.

Key Remote Config changes:

Add RCClientLibConfigManager class for priority-based merging of multiple APM_TRACING configs (Service+Env > Service > Env > Cluster > Org)
Add APM_TRACING_MULTICONFIG, APM_TRACING_ENABLE_DYNAMIC_INSTRUMENTATION, APM_TRACING_ENABLE_CODE_ORIGIN, and APM_TRACING_ENABLE_LIVE_DEBUGGING capability flags
Implement field guarding in config.js #applyRemoteConfig() to adopt highest priority non-null value for each field per RFC
Centralize client library configuration remote config logic in config/remote_config.js

Key Debugger changes:

Add lifecycle methods (isStarted(), stop(), configure()) to debugger/index.js for dynamic control

Motivation

The Debugger (Dynamic Instrumentation/Live Debugging) needs to support Remote Enablement, allowing operators to dynamically enable, disable, or reconfigure the debugger without requiring service restarts.

This implementation follows the APM_TRACING_MULTICONFIG RFC.

codecov · 2025-12-18T07:53:56Z

Codecov Report

❌ Patch coverage is 59.71223% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.68%. Comparing base (408e3f5) to head (f53bcac).

Files with missing lines	Patch %	Lines
packages/dd-trace/src/config/remote_config.js	65.21%	24 Missing ⚠️
packages/dd-trace/src/debugger/index.js	12.00%	22 Missing ⚠️
packages/dd-trace/src/proxy.js	53.84%	6 Missing ⚠️
packages/dd-trace/src/config.js	90.32%	3 Missing ⚠️
packages/dd-trace/src/llmobs/sdk.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                           Coverage Diff                            @@
##           watson/DEBUG-4402/even-more-rc-split    #7137      +/-   ##
========================================================================
- Coverage                                 84.82%   84.68%   -0.15%     
========================================================================
  Files                                       524      524              
  Lines                                     22217    22320     +103     
========================================================================
+ Hits                                      18845    18901      +56     
- Misses                                     3372     3419      +47

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

watson · 2025-12-18T07:54:00Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-12-18T07:54:05Z

Overall package size

Self size: 4.38 MB
Deduped: 5.2 MB
No deduping: 5.2 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.0 | 68.46 kB | 797.03 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

pr-commenter · 2025-12-18T08:06:12Z

Benchmarks

Benchmark execution time: 2025-12-19 20:26:36

Comparing candidate commit f53bcac in PR branch watson/DEBUG-4402/remote-enablement with baseline commit 408e3f5 in branch watson/DEBUG-4402/even-more-rc-split.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 291 metrics, 29 unstable metrics.

Implements Remote Enablement (RE) for Dynamic Instrumentation/Live Debugger via APM_TRACING_MULTICONFIG. This also fixes a long standing issue, now allowing multiple APM_TRACING remote configurations to be received and merged with priority-based rules. Key Remote Config changes: - Add ApmTracingManager class for priority-based merging of multiple APM_TRACING configs (Service+Env > Service > Env > Cluster > Org) - Add APM_TRACING_MULTICONFIG, APM_TRACING_ENABLE_DYNAMIC_INSTRUMENTATION, APM_TRACING_ENABLE_CODE_ORIGIN, and APM_TRACING_ENABLE_LIVE_DEBUGGING capability flags - Implement field guarding in config.js #applyRemote() to adopt highest priority non-null value for each field

shatzi

Nice work, added some comments.

shatzi · 2025-12-20T15:42:36Z

packages/dd-trace/src/config/remote_config.js

+/**
+ * @typedef {object} RemoteConfigOptions
+ * @property {boolean} [dynamic_instrumentation_enabled] - Enable Dynamic Instrumentation
+ * @property {boolean} [live_debugging_enabled] - Enable Live Debugging (deprecated alias)


You didn't add exception_replay field. Because I assume it is not supported yet. Live debugging is also not used as it should enable trigger probes and capture code origin spans variables - see the spec for distributed debugger.

So, for now we only need to look at code origin and dynamic instrumentation flags.

shatzi · 2025-12-20T15:47:50Z

packages/dd-trace/src/config/remote_config.js

+      }
+    }
+
+    log.debug('[config/remote_config] Merged %d configs into lib_config', libConfigCount)


Nit: if all libConfigs were null, we didn't merged anything, should we return {} or null.

Not sure if there is a difference,

shatzi · 2025-12-20T15:54:25Z

packages/dd-trace/src/config/remote_config.js

-  rc.setProductHandler('APM_TRACING', (action, conf) => {
+  // Debugger
+  rc.updateCapabilities(RemoteConfigCapabilities.APM_TRACING_ENABLE_DYNAMIC_INSTRUMENTATION, true)
+  rc.updateCapabilities(RemoteConfigCapabilities.APM_TRACING_ENABLE_LIVE_DEBUGGING, true)


Not sure about this one

shatzi · 2025-12-22T17:19:44Z

packages/dd-trace/src/config/remote_config.js

+
+  const rcClientLibConfigManager = new RCClientLibConfigManager(config.service, config.env)
+
+  rc.setProductHandler('APM_TRACING', (action, conf, configId) => {


does this run on every config we get? this mean are calling config.updateRemoteConfig multiple times. Maybe there is a better way to only call updateRemoteConfig only once we receive all the configs. I assume we would get all of the relevant configs on the first request.

shatzi · 2025-12-22T17:26:38Z

packages/dd-trace/src/proxy.js


        const tracingRemoteConfig = require('./config/remote_config')
-        tracingRemoteConfig.enable(rc, config, this._enableOrDisableTracing.bind(this))
+        tracingRemoteConfig.enable(rc, config, this._updateTracing.bind(this), this._updateDebugger.bind(this))


Why did we choose this design where we pass multiple functions to handle a new config?

I’m hoping we can simplify this so we don’t have to update multiple files every time we add a new product that needs to react to config changes.

Maybe something like a subscriber pattern, or proxy.on('config-change', updateDebugger).

shatzi · 2025-12-22T17:34:46Z

packages/dd-trace/src/llmobs/sdk.js

-    this._config.configure({ ...this._config, llmobs: llmobsConfig })
+    // TODO: It's not nice that it's copying the entire config object to just update the llmobs config.
+    // We should find a better way to do this.
+    this._config.updateOptions({ ...this._config, llmobs: llmobsConfig })


Does the config also include merged values from the remote config?

I’m wondering if there’s a bug where, when we copy the config, merged remote properties get converted into local options properties.

watson self-assigned this Dec 18, 2025

watson added semver-minor debugger Dynamic Instrumentation & Live Debugger labels Dec 18, 2025

This was referenced Dec 18, 2025

refactor(remote-config): replace kPreUpdate with batch handler API (DEBUG-4402) #7121

Open

refactor(remote_config): move appsec-specific logic to appsec folder #7122

Draft

watson mentioned this pull request Dec 18, 2025

refactor(remote_config): move capability registration to feature modules #7127

Draft

This comment has been minimized.

Sign in to view

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 0c5d4ad to 21edda2 Compare December 18, 2025 08:36

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 84300df to 918975b Compare December 18, 2025 08:36

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 21edda2 to 5d6eabd Compare December 18, 2025 09:01

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 918975b to 137abe9 Compare December 18, 2025 10:43

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 5d6eabd to a412829 Compare December 18, 2025 10:43

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 137abe9 to 041ef5b Compare December 18, 2025 11:25

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from a412829 to bc7d261 Compare December 18, 2025 11:25

watson mentioned this pull request Dec 18, 2025

[nodejs] add dynamic detection for Remote Enablement capabilities DataDog/system-tests#5894

Merged

6 tasks

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from bc7d261 to 91ba2c1 Compare December 18, 2025 15:05

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 041ef5b to 573563d Compare December 19, 2025 17:38

watson force-pushed the watson/DEBUG-4402/remote-enablement branch 2 times, most recently from ce9aa71 to c2e520a Compare December 19, 2025 17:44

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 573563d to 152a63b Compare December 19, 2025 18:05

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from c2e520a to f369517 Compare December 19, 2025 18:05

watson force-pushed the watson/DEBUG-4402/even-more-rc-split branch from 152a63b to 408e3f5 Compare December 19, 2025 19:02

watson force-pushed the watson/DEBUG-4402/remote-enablement branch 3 times, most recently from ffd7ab2 to 3d91f59 Compare December 19, 2025 20:05

watson force-pushed the watson/DEBUG-4402/remote-enablement branch from 3d91f59 to f53bcac Compare December 19, 2025 20:18

watson requested review from a team, BridgeAR, shatzi and simon-id December 19, 2025 20:34

shatzi approved these changes Dec 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(debugger): add Remote Enablement support #7137

feat(debugger): add Remote Enablement support #7137

watson commented Dec 18, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 18, 2025 •

edited

Loading

Uh oh!

watson commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

pr-commenter bot commented Dec 18, 2025 •

edited

Loading

Uh oh!

shatzi left a comment

Uh oh!

shatzi Dec 20, 2025

Uh oh!

shatzi Dec 20, 2025

Uh oh!

shatzi Dec 20, 2025

Uh oh!

shatzi Dec 22, 2025

Uh oh!

shatzi Dec 22, 2025

Uh oh!

shatzi Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		const rcClientLibConfigManager = new RCClientLibConfigManager(config.service, config.env)

		rc.setProductHandler('APM_TRACING', (action, conf, configId) => {

feat(debugger): add Remote Enablement support #7137

Are you sure you want to change the base?

feat(debugger): add Remote Enablement support #7137

Conversation

watson commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Uh oh!

codecov bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

watson commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

This comment has been minimized.

pr-commenter bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

shatzi left a comment

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

watson commented Dec 18, 2025 •

edited

Loading

codecov bot commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Dec 18, 2025 •

edited

Loading

pr-commenter bot commented Dec 18, 2025 •

edited

Loading