Are all threads fork safe?

[joshuay03]

Tracer Version(s)

2.18.0

Ruby Version(s)

ruby 3.4.5 (2025-07-16 revision 20cda200d3) +YJIT +PRISM [x86_64-linux]

Relevent Library and Version(s)

No response

Bug Report

We're in the process of enabling Puma's preload_app! config at @buildkite. While auditing threads spawned during boot, Puma flagged a timeout thread that appears to be spawned by a thread in this gem. The callstack only goes so far, but I suspect it's the runtime metrics thread.

callstack

["/usr/local/bundle/gems/timeout-0.4.3/lib/timeout.rb:131:in 'block in Timeout.ensure_timeout_thread_created'",
"/usr/local/bundle/gems/timeout-0.4.3/lib/timeout.rb:129:in 'Thread::Mutex#synchronize'",
"/usr/local/bundle/gems/timeout-0.4.3/lib/timeout.rb:129:in 'Timeout.ensure_timeout_thread_created'",
"/usr/local/bundle/gems/timeout-0.4.3/lib/timeout.rb:180:in 'Timeout.timeout'",
"/usr/local/lib/ruby/3.4.0/net/http.rb:1657:in 'Net::HTTP#connect'",
"/usr/local/lib/ruby/3.4.0/net/http.rb:1636:in 'Net::HTTP#do_start'",
"/usr/local/lib/ruby/3.4.0/net/http.rb:1625:in 'Net::HTTP#start'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/transport/http/adapters/net.rb:39:in 'Datadog::Core::Transport::HTTP::Adapters::Net#open'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/transport/http/adapters/net.rb:77:in 'Datadog::Core::Transport::HTTP::Adapters::Net#post'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/transport/http/adapters/net.rb:44:in 'Datadog::Core::Transport::HTTP::Adapters::Net#call'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/transport/http/api/instance.rb:47:in 'Datadog::Core::Transport::HTTP::API::Instance#call'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:29:in 'block in Datadog::Core::Telemetry::Transport::HTTP::Telemetry::API::Instance#send_telemetry'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/transport/http/api/endpoint.rb:24:in 'Datadog::Core::Transport::HTTP::API::Endpoint#call'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:64:in 'Datadog::Core::Telemetry::Transport::HTTP::Telemetry::API::Endpoint#call'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:40:in 'Datadog::Core::Telemetry::Transport::HTTP::Telemetry::API::Spec#send_telemetry'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:28:in 'Datadog::Core::Telemetry::Transport::HTTP::Telemetry::API::Instance#send_telemetry'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:18:in 'block in Datadog::Core::Telemetry::Transport::HTTP::Telemetry::Client#send_telemetry_payload'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/client.rb:30:in 'Datadog::Core::Telemetry::Transport::HTTP::Client#send_request'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/http/telemetry.rb:17:in 'Datadog::Core::Telemetry::Transport::HTTP::Telemetry::Client#send_telemetry_payload'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/transport/telemetry.rb:43:in 'Datadog::Core::Telemetry::Transport::Telemetry::Transport#send_telemetry'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/emitter.rb:33:in 'Datadog::Core::Telemetry::Emitter#request'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/worker.rb:208:in 'Datadog::Core::Telemetry::Worker#send_event'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/worker.rb:183:in 'block in Datadog::Core::Telemetry::Worker#started!'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/utils/only_once_successful.rb:44:in 'block in Datadog::Core::Utils::OnlyOnceSuccessful#run'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/utils/only_once_successful.rb:41:in 'Thread::Mutex#synchronize'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/utils/only_once_successful.rb:41:in 'Datadog::Core::Utils::OnlyOnceSuccessful#run'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/worker.rb:182:in 'Datadog::Core::Telemetry::Worker#started!'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/telemetry/worker.rb:139:in 'Datadog::Core::Telemetry::Worker#perform'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/queue.rb:16:in 'Datadog::Core::Workers::Queue::PrependedMethods#perform'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/interval_loop.rb:27:in 'block in Datadog::Core::Workers::IntervalLoop::PrependedMethods#perform'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/interval_loop.rb:116:in 'block in Datadog::Core::Workers::IntervalLoop#perform_loop'",
"<internal:kernel>:168:in 'Kernel#loop'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/interval_loop.rb:112:in 'Datadog::Core::Workers::IntervalLoop#perform_loop'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/interval_loop.rb:24:in 'Datadog::Core::Workers::IntervalLoop::PrependedMethods#perform'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/async.rb:27:in 'block in Datadog::Core::Workers::Async::Thread::PrependedMethods#perform'",
"/usr/local/bundle/gems/datadog-2.18.0/lib/datadog/core/workers/async.rb:157:in 'block in Datadog::Core::Workers::Async::Thread#start_worker'"]

When I investigated whether the threads in this gem are fork-safe, I discovered that some are and some aren't, based on their configured fork policy. Additionally, this recent issue suggests that the runtime metrics thread isn't fork-safe, but at the same time, it appears that all of them have been marked as fork-safe for Puma's reporting.

What I'm trying to determine is:

1. Are all threads actually fork-safe?
2. If not, should the non-fork-safe threads be unmarked as such so users are directed here to learn how to make them fork-safe?
3. How can users ensure all threads are fork-safe? Is calling Datadog.configure { } after fork/before boot in the worker still the recommended approach?

Thanks!

Reproduction Code

No response

Configuration Block

No response

Error Logs

No response

Operating System

No response

How does Datadog help you?

No response

[p-datadog]

Hi @joshuay03 ,

it is not the case that all threads are fork-safe.

Datadog.configure { } after a fork should be the correct strategy to ensure the library is correctly working after a fork.

[joshuay03]

it is not the case that all threads are fork-safe.

Datadog.configure { } after a fork should be the correct strategy to ensure the library is correctly working after a fork.

Thanks. In that case, what do you think about addressing:

2. If not, should the non-fork-safe threads be unmarked as such so users are directed here to learn how to make them fork-safe?

I think people rely on Puma's boot report to audit their threads, and I don't think it's a good idea to explicitly exclude threads from that report when they aren't fork safe.

[p-datadog]

There are some nuances to consider because some functionality of dd-trace-rb is only initialized from Rack middleware, for example (hence it should only be initialized in a worker process, not in the parent).

What does preload_app! do exactly? https://www.rubydoc.info/gems/puma/Puma/DSL:preload_app%21 is not very helpful.

In most cases the initialization must be performed for each worker separately (database connections, network connections) because sharing e.g. network sockets between workers results in a broken application. Furthermore, having that initialization done in the parent is usually not productive because the parent won't use those resources, causing those connections etc. to just be wasted.

[joshuay03]

What does preload_app! do exactly? rubydoc.info/gems/puma/Puma/DSL:preload_app%21 is not very helpful.

It loads the application in Puma's master process before its workers are forked in cluster mode. The default behaviour is to load the application in each worker, so enabling this config requires ensuring threads spawned in the master process are re-spawned in the workers.

In most cases the initialization must be performed for each worker separately (database connections, network connections) because sharing e.g. network sockets between workers results in a broken application. Furthermore, having that initialization done in the parent is usually not productive because the parent won't use those resources, causing those connections etc. to just be wasted.

Yes, with Puma it is recommended to to gracefully terminate these things after the app has been loaded in the master process. So you'd do something like:

# config/puma.rb

# When preloading, the app has been loaded by this point
before_fork do
  StatsdConnection.close_all # Datadog::Statsd singleton
end

# When preloading, the loaded app is available in the worker by this point
on_worker_boot do
  # Nothing to do here because the next time a StatsdConnection is required,
  # the singleton will lazily initialise one in this child process.
  #
  # If that is not the case however, usually you would eager establish connections etc. here.
end

on_worker_shutdown do
  StatsdConnection.flush_all(sync: true)
end

A few gems however handle all this for you e.g. activerecord. See:

• https://github.com/rails/rails/blob/b8278772601cecdf7d4b76a753a43db39e5ef3aa/activerecord/lib/active_record/connection_adapters/pool_config.rb#L83
• https://github.com/rails/rails/blob/b8278772601cecdf7d4b76a753a43db39e5ef3aa/activerecord/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb#L44

which is why they mark themselves as :fork_safe for Puma.

[p-datadog]

We have the fork policy for telemetry worker set to stop in https://github.com/DataDog/dd-trace-rb/blob/master/lib/datadog/core/telemetry/worker.rb#L43 and https://github.com/DataDog/dd-trace-rb/blob/master/lib/datadog/core/configuration/components.rb#L136 starts telemetry whenever the component tree is created, meaning telemetry worker thread won't exist in any fork. I am following up to see if we can figure out the reasoning for this behavior.

[p-datadog]

We don't seem to have a solid reason why the current behavior is what it is, personally I am of the opinion that telemetry should restart its threads after forking.

[joshuay03]

We don't seem to have a solid reason why the current behavior is what it is, personally I am of the opinion that telemetry should restart its threads after forking.

Yes that would be great. If that's going to take a while however, please consider un-marking it as fork safe for the time being. I also opened DataDog/dogstatsd-ruby#315 to do the opposite in the dogstatsd-ruby gem.