[INCIDENT-46145] Rewrite security response tests #5083

Strech · 2025-11-24T13:41:57Z

What does this PR do?

In this PR I've rewritten a bit outdated tests and fix a potential flakiness on JRuby when we write in a file and reconfigure global configuration state with after block in shared example.

      shared_examples_for 'with custom response body' do |type|
        before do
          File.write("test.#{type}", 'testing')
          Datadog.configuration.appsec.block.templates.send("#{type}=", "test.#{type}")
        end

        after do
          File.delete("test.#{type}")
          Datadog.configuration.appsec.reset!
        end

        it { is_expected.to eq ['testing'] }
      end

Motivation:

Outdated tests have some issues on JRuby on some random occasion. Let's streamline the tests and also reduce flakiness (I hope 🤞🏼)

Change log entry

No.

Additional Notes:

No.

How to test the change?

CI.

pr-commenter · 2025-11-24T14:34:19Z

Benchmarks

Benchmark execution time: 2025-11-24 19:39:17

Comparing candidate commit e63a425 in PR branch fix-jruby-appsec-response-spec with baseline commit 57aa262 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 2 unstable metrics.

y9v

great job! It's much more readable now

lloeki

It failed on 3.4 though.

Very odd that this doesn't make the merge button red so I'm putting a request changes to prevent an accidental merge.

Failures:

  1) Datadog::AppSec::Response.from_interrupt_params .body when Accept header value is application/json returns default json template with security response ID
     Failure/Error:
       expect(response.body[0]).to include(
         /{"errors":.*,"security_response_id":"00000000-0000-0000-0000-000000000000".*}/
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:125:in 'block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

  2) Datadog::AppSec::Response.from_interrupt_params .body when Accept header value is text/html returns HTML template with security response ID
     Failure/Error:
       expect(response.body[0]).to include(
         /.*<!DOCTYPE html>\n.*<p class="security-response-id">.*: 00000000-0000-0000-0000-000000000000/m
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:111:in 'block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

  3) Datadog::AppSec::Response.from_interrupt_params .body when Accept header value is not supported returns default json template with security response ID
     Failure/Error:
       expect(response.body[0]).to include(
         /{"errors":.*,"security_response_id":"00000000-0000-0000-0000-000000000000".*}/
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:97:in 'block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

  4) Datadog::AppSec::Response.from_interrupt_params .body when default template is changed to custom returns custom template with security response ID
     Failure/Error:
       expect(response.body[0]).to include(
         /Blocked, that's an ID: 00000000-0000-0000-0000-000000000000/
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:172:in 'block (5 levels) in <top (required)>'
     # ./spec/datadog/appsec/response_spec.rb:159:in 'block (6 levels) in <top (required)>'
     # ./spec/datadog/appsec/response_spec.rb:147:in 'block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

  5) Datadog::AppSec::Response.from_interrupt_params .body when Accept header value is text/plain returns default json template with security response ID
     Failure/Error:
       expect(response.body[0]).to include(
         /You've been blocked.*Security Response ID: 00000000-0000-0000-0000-000000000000.*/m
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:139:in 'block (5 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

  6) Datadog::AppSec::Response.from_interrupt_params when response is a block response returns response with block attributes
     Failure/Error:
       expect(response.body[0]).to include(
         /<p class="security-response-id">.*: 00000000-0000-0000-0000-000000000000/
       )

     TypeError:
       no implicit conversion of Regexp into String
     # ./spec/datadog/appsec/response_spec.rb:74:in 'block (4 levels) in <top (required)>'
     # ./spec/spec_helper.rb:274:in 'block (2 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'
     # ./spec/support/execute_in_fork.rb:32:in 'ForkableExample#run'

Finished in 1 minute 53.24 seconds (files took 2.64 seconds to load)
5657 examples, 6 failures, 62 pending

datadog-official · 2025-11-24T19:29:03Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage
• Patch Coverage: 98.73%
• Total Coverage: 95.17% (-0.00%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: e63a425 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

doesn't make sense

Strech requested a review from a team as a code owner November 24, 2025 13:41

github-actions bot added the dev/testing Involves testing processes (e.g. RSpec) label Nov 24, 2025

Strech force-pushed the fix-jruby-appsec-response-spec branch from 296e158 to 666b5a3 Compare November 24, 2025 14:03

Strech force-pushed the fix-jruby-appsec-response-spec branch from 666b5a3 to 6a641c0 Compare November 24, 2025 14:43

y9v approved these changes Nov 24, 2025

View reviewed changes

lloeki previously requested changes Nov 24, 2025

View reviewed changes

Rewrite security response tests

e63a425

Strech force-pushed the fix-jruby-appsec-response-spec branch from 6a641c0 to e63a425 Compare November 24, 2025 19:08

Strech merged commit 7084871 into master Nov 25, 2025
266 checks passed

Strech deleted the fix-jruby-appsec-response-spec branch November 25, 2025 09:08

github-actions bot added this to the 2.23.0 milestone Nov 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[INCIDENT-46145] Rewrite security response tests #5083

[INCIDENT-46145] Rewrite security response tests #5083

Strech commented Nov 24, 2025

Uh oh!

pr-commenter bot commented Nov 24, 2025 •

edited

Loading

Uh oh!

y9v left a comment

Uh oh!

lloeki left a comment

Uh oh!

datadog-official bot commented Nov 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[INCIDENT-46145] Rewrite security response tests #5083

[INCIDENT-46145] Rewrite security response tests #5083

Conversation

Strech commented Nov 24, 2025

Uh oh!

pr-commenter bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

y9v left a comment

Choose a reason for hiding this comment

Uh oh!

lloeki left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-official bot commented Nov 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pr-commenter bot commented Nov 24, 2025 •

edited

Loading