Skip to content

Conversation

@ivoanjo
Copy link
Member

@ivoanjo ivoanjo commented Feb 10, 2025

What does this PR do?

This PR fixes the following flaky profiler spec
(seen in
https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/19126/workflows/00361a70-d216-4fa9-9ce9-0f87e5f033f1/jobs/669239):

Failures:

  1) Datadog::Profiling::Collectors::CpuAndWallTimeWorker._native_allocation_count when CpuAndWallTimeWorker has been started when allocation profiling and allocation counting is enabled returns different numbers of allocations for different threads
     Failure/Error: expect(after_t1 - before_t1).to be 100

       expected #<Integer:201> => 100
            got #<Integer:217> => 108

       Compared using equal?, which compares object identity,
       but expected and actual are not the same object. Use
       `expect(actual).to eq(expected)` if you don't care about
       object identity in this example.
     # ./spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:1377:in `block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:238:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:123:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block (2 levels) in <top (required)>'

We had previously debugged the same issue on another spec in #3554 (and left a nice comment explaining what the issue is).

I just propagated the same fix (disabling GC for the duration of the test) to this spec as well.

Motivation:

Zero known flaky profiler tests!

Change log entry

Nope

Additional Notes:

In practice what happens is that we're counting how many Ruby objects get created during the spec, but if GC gets triggered at the exact wrong time, it may trigger other Ruby code running, and thus more objects than expected being allocated.

By disabling GC for this test, hopefully we're able to keep the Ruby VM from not allocating any incidental things in the background.

This GC counting feature is also deprecated, so in the future we'll remove it which is even nicer :)

How to test the change?

Validate that CI is still green.

**What does this PR do?**

This PR fixes the following flaky profiler spec
(seen in
https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/19126/workflows/00361a70-d216-4fa9-9ce9-0f87e5f033f1/jobs/669239):

```
Failures:

  1) Datadog::Profiling::Collectors::CpuAndWallTimeWorker._native_allocation_count when CpuAndWallTimeWorker has been started when allocation profiling and allocation counting is enabled returns different numbers of allocations for different threads
     Failure/Error: expect(after_t1 - before_t1).to be 100

       expected #<Integer:201> => 100
            got #<Integer:217> => 108

       Compared using equal?, which compares object identity,
       but expected and actual are not the same object. Use
       `expect(actual).to eq(expected)` if you don't care about
       object identity in this example.
     # ./spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:1377:in `block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:238:in `block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:123:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block (2 levels) in <top (required)>'
```

We had previously debugged the same issue on another spec in
#3554 (and left a nice
comment explaining what the issue is).

I just propagated the same fix (disabling GC for the duration
of the test) to this spec as well.

**Motivation:**

Zero known flaky profiler tests!

**Additional Notes:**

In practice what happens is that we're counting how many
Ruby objects get created during the spec, but if GC gets
triggered at the exact wrong time, it may trigger other
Ruby code running, and thus more objects than expected
being allocated.

By disabling GC for this test, hopefully we're able
to keep the Ruby VM from not allocating any incidental
things in the background.

This GC counting feature is also deprecated, so in the
future we'll remove it which is even nicer :)

**How to test the change?**

Validate that CI is still green.
@ivoanjo ivoanjo added dev/testing Involves testing processes (e.g. RSpec) profiling Involves Datadog profiling labels Feb 10, 2025
@ivoanjo ivoanjo requested review from a team as code owners February 10, 2025 17:15
@datadog-datadog-prod-us1
Copy link
Contributor

datadog-datadog-prod-us1 bot commented Feb 10, 2025

Datadog Report

Branch report: ivoanjo/fix-profiling-flaky-spec
Commit report: 8f52c60
Test service: dd-trace-rb

✅ 0 Failed, 22040 Passed, 1478 Skipped, 5m 53.1s Total Time

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.72%. Comparing base (fba70fd) to head (8f52c60).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4362      +/-   ##
==========================================
- Coverage   97.73%   97.72%   -0.01%     
==========================================
  Files        1347     1347              
  Lines       82388    82389       +1     
  Branches     4193     4193              
==========================================
- Hits        80522    80518       -4     
- Misses       1866     1871       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pr-commenter
Copy link

pr-commenter bot commented Feb 10, 2025

Benchmarks

Benchmark execution time: 2025-02-10 18:28:08

Comparing candidate commit 8f52c60 in PR branch ivoanjo/fix-profiling-flaky-spec with baseline commit fba70fd in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 31 metrics, 2 unstable metrics.

@ivoanjo ivoanjo enabled auto-merge February 11, 2025 09:24
@ivoanjo ivoanjo merged commit 73879c0 into master Feb 11, 2025
286 checks passed
@ivoanjo ivoanjo deleted the ivoanjo/fix-profiling-flaky-spec branch February 11, 2025 09:29
@github-actions github-actions bot added this to the 2.11.0 milestone Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev/testing Involves testing processes (e.g. RSpec) profiling Involves Datadog profiling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants