DEBUG-3700 Do not initialize components from crash tracking on fork handler #4619

p-datadog · 2025-05-01T20:36:37Z

What does this PR do?

Prevents crash tracker from initializing the component tree in its on fork handler

Motivation:

When running tests, the component tree may not be initialized, but the on fork handler in crash tracker may be active. This causes crash tracker to attempt to initialize component tree, interfering with the test being run. For example, some of the telemetry tests execute assertions in a fork, which causes crash tracker's on fork handler to run, which then creates another telemetry component instance (with wrong settings, since the correct settings were specified in the unit test and not via a global Datadog.configure), which then interferes with the test's assertions.

For example, running the following tests fails locally:

rspec  spec/datadog/core/telemetry/event_spec.rb spec/datadog/core/telemetry/component_spec.rb --seed 1

With the following errors repeated many times over:

Failures:

  1) Datadog::Core::Telemetry::Component#client_configuration_change! when in fork 
     Failure/Error: stdout ||= File.read(fork_stdout.path)
     
     Errno::ENOENT:
       No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-b9rd3y
     # ./spec/support/synchronization_helpers.rb:39:in `read'
     # ./spec/support/synchronization_helpers.rb:39:in `rescue in expect_in_fork'
     # ./spec/support/synchronization_helpers.rb:38:in `expect_in_fork'
     # ./spec/datadog/core/telemetry/component_spec.rb:224:in `block (4 levels) in <top (required)>'
     # ./spec/spec_helper.rb:263:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:148:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/webmock-3.25.1/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ENOENT:
     #   No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-b9rd3y
     #   ./spec/support/synchronization_helpers.rb:28:in `read'

  2) Datadog::Core::Telemetry::Component#integrations_change! when in fork 
     Failure/Error: stdout ||= File.read(fork_stdout.path)
     
     Errno::ENOENT:
       No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-6oa51s
     # ./spec/support/synchronization_helpers.rb:39:in `read'
     # ./spec/support/synchronization_helpers.rb:39:in `rescue in expect_in_fork'
     # ./spec/support/synchronization_helpers.rb:38:in `expect_in_fork'
     # ./spec/datadog/core/telemetry/component_spec.rb:184:in `block (4 levels) in <top (required)>'
     # ./spec/spec_helper.rb:263:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:148:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/webmock-3.25.1/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ENOENT:
     #   No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-6oa51s
     #   ./spec/support/synchronization_helpers.rb:28:in `read'

  3) Datadog::Core::Telemetry::Component#emit_closing! when in fork 
     Failure/Error: stdout ||= File.read(fork_stdout.path)
     
     Errno::ENOENT:
       No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-lu56oy
     # ./spec/support/synchronization_helpers.rb:39:in `read'
     # ./spec/support/synchronization_helpers.rb:39:in `rescue in expect_in_fork'
     # ./spec/support/synchronization_helpers.rb:38:in `expect_in_fork'
     # ./spec/datadog/core/telemetry/component_spec.rb:134:in `block (4 levels) in <top (required)>'
     # ./spec/spec_helper.rb:263:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:148:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/webmock-3.25.1/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ENOENT:
     #   No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-lu56oy
     #   ./spec/support/synchronization_helpers.rb:28:in `read'

  4) Datadog::Core::Telemetry::Component#log! when enabled and log_collection_enabled is enabled when in fork 
     Failure/Error: stdout ||= File.read(fork_stdout.path)
     
     Errno::ENOENT:
       No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-te7w89
     # ./spec/support/synchronization_helpers.rb:39:in `read'
     # ./spec/support/synchronization_helpers.rb:39:in `rescue in expect_in_fork'
     # ./spec/support/synchronization_helpers.rb:38:in `expect_in_fork'
     # ./spec/datadog/core/telemetry/component_spec.rb:258:in `block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:263:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:148:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/webmock-3.25.1/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /home/w/.cache/vendor/bundle/ruby/3.3.0/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ENOENT:
     #   No such file or directory @ rb_sysopen - /tmp/datadog-rspec-expect-in-fork-stdout20250512-46944-te7w89
     #   ./spec/support/synchronization_helpers.rb:28:in `read'

Finished in 3.41 seconds (files took 0.48067 seconds to load)
56 examples, 4 failures

Failed examples:

rspec ./spec/datadog/core/telemetry/component_spec.rb:222 # Datadog::Core::Telemetry::Component#client_configuration_change! when in fork 
rspec ./spec/datadog/core/telemetry/component_spec.rb:182 # Datadog::Core::Telemetry::Component#integrations_change! when in fork 
rspec ./spec/datadog/core/telemetry/component_spec.rb:132 # Datadog::Core::Telemetry::Component#emit_closing! when in fork 
rspec ./spec/datadog/core/telemetry/component_spec.rb:256 # Datadog::Core::Telemetry::Component#log! when enabled and log_collection_enabled is enabled when in fork

Change log entry

None

Additional Notes:

This PR also stops duplicate reports of the test failures when expect_in_fork is used. For example, the above two tests have 4 failures and those 4 failures are printed a total of 16 times instead of once.

How to test the change?
Existing CI

github-actions · 2025-05-01T20:36:54Z

Thank you for updating Change log entry section 👏

^{Visited at: 2025-05-12 18:59:02 UTC}

codecov-commenter · 2025-05-01T21:05:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.76%. Comparing base (f927b70) to head (847ceb2).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4619      +/-   ##
==========================================
- Coverage   97.77%   97.76%   -0.02%     
==========================================
  Files        1418     1418              
  Lines       86587    86586       -1     
  Branches     4396     4396              
==========================================
- Hits        84662    84647      -15     
- Misses       1925     1939      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-datadog-prod-us1 · 2025-05-01T21:05:36Z

Datadog Report

Branch report: crashtracking-component-tree
Commit report: 847ceb2
Test service: dd-trace-rb

✅ 0 Failed, 21441 Passed, 1383 Skipped, 3m 55.76s Total Time

pr-commenter · 2025-05-01T21:16:32Z

Benchmarks

Benchmark execution time: 2025-05-14 14:44:09

Comparing candidate commit 847ceb2 in PR branch crashtracking-component-tree with baseline commit f927b70 in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 30 metrics, 2 unstable metrics.

scenario:profiling - Allocations (profiling enabled)

🟩 throughput [+246543.422op/s; +256481.928op/s] or [+5.120%; +5.327%]

ivoanjo · 2025-05-02T07:50:51Z

I'm curious what drove this change (I checked the JIRA ticket and didn't have any details about this).

In particular, the at_fork in the crashtracker gets applied once the crashtracker is initialized by the components, so I don't understand in what situation we'd go back to a case where the components are not initialized again after that?

ivoanjo

👍 Thanks for adding the extra context to understand this.

It sucks a bit to be changing the production code to accommodate something that only happens in testing, but we have tests that cover the behavior in forks, so this should be fine.

github-actions bot added the core Involves Datadog core libraries label May 1, 2025

p-datadog changed the title ~~Do not initialize components from on crashtracking fork handler~~ DEBUG-3700 Do not initialize components from on crashtracking fork handler May 1, 2025

do not initialize components from on fork handler

dfb0d36

p-datadog force-pushed the crashtracking-component-tree branch from ad849bc to dfb0d36 Compare May 7, 2025 21:51

p-datadog changed the title ~~DEBUG-3700 Do not initialize components from on crashtracking fork handler~~ DEBUG-3700 Do not initialize components from crash tracking on fork handler May 12, 2025

p-datadog mentioned this pull request May 12, 2025

DEBUG-3700 Inject telemetry into metrics client #4641

Merged

p-datadog marked this pull request as ready for review May 13, 2025 12:08

p-datadog requested a review from a team as a code owner May 13, 2025 12:08

Merge branch 'master' into crashtracking-component-tree

991446a

ivoanjo approved these changes May 14, 2025

View reviewed changes

Merge branch 'master' into crashtracking-component-tree

847ceb2

p-datadog merged commit a8c8128 into master May 14, 2025
1048 of 1054 checks passed

p-datadog deleted the crashtracking-component-tree branch May 14, 2025 23:08

github-actions bot added this to the 2.16.0 milestone May 14, 2025

p-datadog mentioned this pull request May 28, 2025

DEBUG-3700 more explicitly access active remote #4642

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

DEBUG-3700 Do not initialize components from crash tracking on fork handler #4619

DEBUG-3700 Do not initialize components from crash tracking on fork handler #4619

Uh oh!

p-datadog commented May 1, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 1, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented May 1, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented May 1, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 1, 2025 •

edited

Loading

Uh oh!

ivoanjo commented May 2, 2025

Uh oh!

ivoanjo left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DEBUG-3700 Do not initialize components from crash tracking on fork handler #4619

DEBUG-3700 Do not initialize components from crash tracking on fork handler #4619

Uh oh!

Conversation

p-datadog commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-datadog-prod-us1 bot commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

pr-commenter bot commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:profiling - Allocations (profiling enabled)

Uh oh!

ivoanjo commented May 2, 2025

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

p-datadog commented May 1, 2025 •

edited

Loading

github-actions bot commented May 1, 2025 •

edited

Loading

codecov-commenter commented May 1, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented May 1, 2025 •

edited

Loading

pr-commenter bot commented May 1, 2025 •

edited

Loading