-
Notifications
You must be signed in to change notification settings - Fork 399
[NO-TICKET] Fix flaky profiler object counting spec #4362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
**What does this PR do?** This PR fixes the following flaky profiler spec (seen in https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/19126/workflows/00361a70-d216-4fa9-9ce9-0f87e5f033f1/jobs/669239): ``` Failures: 1) Datadog::Profiling::Collectors::CpuAndWallTimeWorker._native_allocation_count when CpuAndWallTimeWorker has been started when allocation profiling and allocation counting is enabled returns different numbers of allocations for different threads Failure/Error: expect(after_t1 - before_t1).to be 100 expected #<Integer:201> => 100 got #<Integer:217> => 108 Compared using equal?, which compares object identity, but expected and actual are not the same object. Use `expect(actual).to eq(expected)` if you don't care about object identity in this example. # ./spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:1377:in `block (5 levels) in <top (required)>' # ./spec/spec_helper.rb:238:in `block (2 levels) in <top (required)>' # ./spec/spec_helper.rb:123:in `block (2 levels) in <top (required)>' # /usr/local/bundle/gems/webmock-3.23.0/lib/webmock/rspec.rb:39:in `block (2 levels) in <top (required)>' # /usr/local/bundle/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block (2 levels) in <top (required)>' ``` We had previously debugged the same issue on another spec in #3554 (and left a nice comment explaining what the issue is). I just propagated the same fix (disabling GC for the duration of the test) to this spec as well. **Motivation:** Zero known flaky profiler tests! **Additional Notes:** In practice what happens is that we're counting how many Ruby objects get created during the spec, but if GC gets triggered at the exact wrong time, it may trigger other Ruby code running, and thus more objects than expected being allocated. By disabling GC for this test, hopefully we're able to keep the Ruby VM from not allocating any incidental things in the background. This GC counting feature is also deprecated, so in the future we'll remove it which is even nicer :) **How to test the change?** Validate that CI is still green.
Datadog ReportBranch report: ✅ 0 Failed, 22040 Passed, 1478 Skipped, 5m 53.1s Total Time |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4362 +/- ##
==========================================
- Coverage 97.73% 97.72% -0.01%
==========================================
Files 1347 1347
Lines 82388 82389 +1
Branches 4193 4193
==========================================
- Hits 80522 80518 -4
- Misses 1866 1871 +5 ☔ View full report in Codecov by Sentry. |
BenchmarksBenchmark execution time: 2025-02-10 18:28:08 Comparing candidate commit 8f52c60 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 31 metrics, 2 unstable metrics. |
What does this PR do?
This PR fixes the following flaky profiler spec
(seen in
https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/19126/workflows/00361a70-d216-4fa9-9ce9-0f87e5f033f1/jobs/669239):
We had previously debugged the same issue on another spec in #3554 (and left a nice comment explaining what the issue is).
I just propagated the same fix (disabling GC for the duration of the test) to this spec as well.
Motivation:
Zero known flaky profiler tests!
Change log entry
Nope
Additional Notes:
In practice what happens is that we're counting how many Ruby objects get created during the spec, but if GC gets triggered at the exact wrong time, it may trigger other Ruby code running, and thus more objects than expected being allocated.
By disabling GC for this test, hopefully we're able to keep the Ruby VM from not allocating any incidental things in the background.
This GC counting feature is also deprecated, so in the future we'll remove it which is even nicer :)
How to test the change?
Validate that CI is still green.