Skip to content

Commit 8650e35

Browse files
ivoanjoivoanjo
committed
[NO-TICKET] Workaround profiling benchmark flakiness on Ruby 2.6
**What does this PR do?** This PR skips the `profiling/validate_benchmarks_spec.rb` on Ruby 2.6 as these have proven flaky and we've not been able to figure out why yet. The objective of this spec is to validate that the benchmarks don't bitrot -- nothing else, so for now let's skip running them on Ruby 2.6 and rely on the fact they do run on other Ruby versions to keep them working (and maybe someday get to the bottom of this issue). This removes this issue as a common source of CI flakiness. **Motivation:** For a while we've been hunting some flakiness in the profiling benchmarks that seems to especially affect Ruby 2.6 -- see #4947 . The mechanism we added on that PR didn't particularly help diagnose what's going on: ``` 1) Profiling benchmarks profiling_sample_loop_v2 runs without raising errors Failure/Error: raise "Failure or timeout in `expect_in_fork`#{crash_note}, STDOUT: `#{stdout}`, STDERR: `#{stderr}`", cause: e RuntimeError: Failure or timeout in `expect_in_fork` (Crashing Ruby to get stacktrace as requested by `trigger_stacktrace_on_kill`), STDOUT: `Current pid is 4727 Calculating ------------------------------------- profiling - stack collector (ruby frames - native filenames enabled) 4.240k (±13.4%) i/s - 39.000 in 0.010075s `, STDERR: `/__w/dd-trace-rb/dd-trace-rb/benchmarks/profiling_sample_loop_v2.rb:16: [BUG] Segmentation fault at 0x000000000000124f ruby 2.6.10p210 (2022-04-12 revision 67958) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0004 p:---- s:0014 e:000013 CFUNC :sleep c:0003 p:0005 s:0010 e:000009 BLOCK /__w/dd-trace-rb/dd-trace-rb/benchmarks/profiling_sample_loop_v2.rb:16 c:0002 p:0011 s:0007 e:000006 BLOCK /__w/dd-trace-rb/dd-trace-rb/spec/spec_helper.rb:290 [FINISH] c:0001 p:---- s:0003 e:000002 (none) [FINISH] -- Ruby level backtrace information ---------------------------------------- /__w/dd-trace-rb/dd-trace-rb/spec/spec_helper.rb:290:in `block in initialize' /__w/dd-trace-rb/dd-trace-rb/benchmarks/profiling_sample_loop_v2.rb:16:in `block in <class:ProfilerSampleLoopBenchmark>' /__w/dd-trace-rb/dd-trace-rb/benchmarks/profiling_sample_loop_v2.rb:16:in `sleep' -- Machine register context ------------------------------------------------ RIP: 0x00007f7d3169bb16 RBP: 0x00005613aa1949e0 RSP: 0x00007ffc8b1f59a0 RAX: 0xfffffffffffffffc RBX: 0x00005613aa194a08 RCX: 0x00007f7d3169bb16 RDX: 0x0000000000000000 RDI: 0x00007ffc8b1f5a40 RSI: 0x0000000000000002 R8: 0x0000000000000008 R9: 0x0000000000000000 R10: 0x0000000000000000 R11: 0x0000000000000293 R12: 0x0000000000000000 R13: 0x00005613b05201f0 R14: 0x0000000000000000 R15: 0x00007f7d3106fec8 EFL: 0x0000000000000293 -- C level backtrace information ------------------------------------------- /usr/local/lib/libruby.so.2.6(rb_print_backtrace+0x11) [0x7f7d319f742a] vm_dump.c:715 /usr/local/lib/libruby.so.2.6(rb_vm_bugreport) vm_dump.c:985 /usr/local/lib/libruby.so.2.6(rb_bug_context+0xec) [0x7f7d31847cec] error.c:609 /usr/local/lib/libruby.so.2.6(sigsegv+0x42) [0x7f7d3195e752] signal.c:998 /usr/local/bundle/gems/libdatadog-24.0.1.1.0-x86_64-linux/vendor/libdatadog-24.0.1/x86_64-linux/libdatadog-x86_64-unknown-linux-gnu/lib/pkgconfig/../../lib/libdatadog_profiling.so(_ZN18libdd_crashtracker9collector13crash_handler22handle_posix_sigaction17hccec31010ba0e887E+0x59e) [0x7f7d2dfc2110] /lib/x86_64-linux-gnu/libc.so.6(0x7f7d315e4de0) [0x7f7d315e4de0] /lib/x86_64-linux-gnu/libc.so.6(ppoll+0xa6) [0x7f7d3169bb16] /usr/local/lib/libruby.so.2.6(native_ppoll_sleep+0x19a) [0x7f7d319a059a] thread_pthread.c:2131 /usr/local/lib/libruby.so.2.6(native_sleep+0x2fa) [0x7f7d319a10ea] thread_pthread.c:2170 /usr/local/lib/libruby.so.2.6(thread_join_sleep+0x12f) [0x7f7d319a454f] thread.c:987 /usr/local/lib/libruby.so.2.6(rb_ensure+0xce) [0x7f7d3184fa3e] eval.c:1088 /usr/local/lib/libruby.so.2.6(thread_join+0x38) [0x7f7d3199d178] thread.c:1032 /usr/local/lib/libruby.so.2.6(thread_join_m) thread.c:1129 /usr/local/lib/libruby.so.2.6(vm_call0_cfunc_with_frame+0x85) [0x7f7d319e85cf] vm_eval.c:86 /usr/local/lib/libruby.so.2.6(vm_call0_cfunc) vm_eval.c:100 /usr/local/lib/libruby.so.2.6(vm_call0_body) vm_eval.c:131 /usr/local/lib/libruby.so.2.6(rb_vm_call0) vm_eval.c:60 /usr/local/lib/libruby.so.2.6(invoke_block_from_c_bh+0x1f) [0x7f7d319f18ef] vm_insnhelper.c:2555 /usr/local/lib/libruby.so.2.6(vm_yield) vm.c:1155 /usr/local/lib/libruby.so.2.6(rb_yield_0) vm_eval.c:978 /usr/local/lib/libruby.so.2.6(rb_yield_1) vm_eval.c:984 /usr/local/lib/libruby.so.2.6(rb_yield) vm_eval.c:994 /usr/local/lib/libruby.so.2.6(rb_array_len+0x0) [0x7f7d317be1ac] array.c:2087 /usr/local/lib/libruby.so.2.6(rb_ary_each) array.c:2086 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(rb_load_internal0+0xff) [0x7f7d3189be8f] load.c:612 /usr/local/lib/libruby.so.2.6(rb_load_internal+0xb) [0x7f7d3189c293] load.c:643 /usr/local/lib/libruby.so.2.6(rb_f_load) load.c:711 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x148) [0x7f7d319dea38] insns.def:765 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(rb_protect+0x1e5) [0x7f7d3184f7f5] eval.c:1046 /usr/local/lib/libruby.so.2.6(rb_f_fork+0x16) [0x7f7d3191502f] process.c:4139 /usr/local/lib/libruby.so.2.6(rb_f_fork) process.c:4130 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x1e05) [0x7f7d319e06f5] insns.def:836 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(yield_under+0x3e6) [0x7f7d319f2b56] vm.c:1021 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x856) [0x7f7d319e4d66] vm.c:1894 /usr/local/lib/libruby.so.2.6(rb_yield+0x276) [0x7f7d319f15e6] vm.c:1021 /usr/local/lib/libruby.so.2.6(rb_ary_collect+0x5c) [0x7f7d317c3dac] array.c:3017 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(rb_yield+0x276) [0x7f7d319f15e6] vm.c:1021 /usr/local/lib/libruby.so.2.6(rb_ary_collect+0x5c) [0x7f7d317c3dac] array.c:3017 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(rb_yield+0x276) [0x7f7d319f15e6] vm.c:1021 /usr/local/lib/libruby.so.2.6(rb_ary_collect+0x5c) [0x7f7d317c3dac] array.c:3017 /usr/local/lib/libruby.so.2.6(vm_cfp_consistent_p+0x0) [0x7f7d319d415b] vm_insnhelper.c:1908 /usr/local/lib/libruby.so.2.6(vm_call_cfunc_with_frame) vm_insnhelper.c:1910 /usr/local/lib/libruby.so.2.6(vm_call_cfunc) vm_insnhelper.c:1924 /usr/local/lib/libruby.so.2.6(vm_exec_core+0x234) [0x7f7d319deb24] insns.def:750 /usr/local/lib/libruby.so.2.6(rb_vm_exec+0x155) [0x7f7d319e4665] vm.c:1885 /usr/local/lib/libruby.so.2.6(ruby_exec_internal+0xc4) [0x7f7d3184c4a4] eval.c:262 /usr/local/lib/libruby.so.2.6(ruby_exec_node+0x1d) [0x7f7d3184e96d] eval.c:326 /usr/local/lib/libruby.so.2.6(ruby_run_node+0x2e) [0x7f7d3185161e] eval.c:318 /usr/local/bin/ruby(main+0x5b) [0x56137eb1310b] ./main.c:42 ``` **Additional Notes:** N/A **How to test the change?** Validate spec still runs on all Rubies other than 2.6.
1 parent 2c84fc0 commit 8650e35

File tree

1 file changed

+18
-1
lines changed

1 file changed

+18
-1
lines changed

‎spec/datadog/profiling/validate_benchmarks_spec.rb‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,24 @@
11
require "datadog/profiling/spec_helper"
22

33
RSpec.describe "Profiling benchmarks", :memcheck_valgrind_skip do
4-
before { skip_if_profiling_not_supported(self) }
4+
before do
5+
skip_if_profiling_not_supported(self)
6+
7+
# @ivoanjo: We've seen these tests be flaky especially on Ruby 2.6: https://github.com/DataDog/dd-trace-rb/pull/4947
8+
# Even when trying to get a backtrace out of Ruby, all we saw was one of the Ruby threads sleeping
9+
# e.g. https://github.com/DataDog/dd-trace-rb/actions/runs/19484921634/job/55765012312#step:5:8980 :
10+
#
11+
# -- Control frame information -----------------------------------------------
12+
# c:0004 p:---- s:0014 e:000013 CFUNC :sleep
13+
# c:0003 p:0005 s:0010 e:000009 BLOCK /__w/dd-trace-rb/dd-trace-rb/benchmarks/profiling_sample_loop_v2.rb:16
14+
# c:0002 p:0011 s:0007 e:000006 BLOCK /__w/dd-trace-rb/dd-trace-rb/spec/spec_helper.rb:290 [FINISH]
15+
# c:0001 p:---- s:0003 e:000002 (none) [FINISH]
16+
#
17+
# Because the objective of this spec is making sure the benchmarks don't bitrot, and seeing as this flakiness seems
18+
# to especially affect 2.6 but we have no indication otherwise that we have issues in Ruby 2.6, I decided to skip
19+
# this for now and rely on this spec running on other Ruby versions to validate the benchmark is ok.
20+
skip("Skipping on Ruby 2.6 as it's flaky and we couldn't figure out why yet") if RUBY_VERSION.start_with?("2.6")
21+
end
522

623
around do |example|
724
ClimateControl.modify("VALIDATE_BENCHMARK" => "true") do

0 commit comments

Comments
 (0)