Possible memory leaks with AsyncAutoResetEvent

[KostyaLocal]

use case:
Class MyClass : BackgroundService {
private readonly AsyncAutoResetEvent _wakeEvent = new(false);
protected override async Task ExecuteAsync(CancellationToken ct) {
const int delay = 100;
var timer = Stopwatch.StartNew();
while (!ct.IsCancellationRequested) {
var remaining = (double)Math.Max(0, delay - timer.ElapsedMilliseconds);
await _wakeEvent.WaitAsync(TimeSpan.FromMilliseconds(remaining), CancellationToken.None);
timer.Restart();
}
}

[sakno]

What are the symptoms for the possible memory leak?

That's the rewritten example:

[Fact]
public static async Task MemoryLeak()
{
    using AsyncAutoResetEvent wakeEvent = new(false);
    
    const int delay = 100;
    var timer = Stopwatch.StartNew();
    for (var i = 0; i < 100; i++) {
        var remaining = (double)Math.Max(0, delay - timer.ElapsedMilliseconds);
        await wakeEvent.WaitAsync(TimeSpan.FromMilliseconds(remaining), CancellationToken.None);
        timer.Restart();
    }

    // GC.Collect();
    // GC.WaitForPendingFinalizers();
    
    Debugger.Break();
}

WaitAsync produces a new instance of CancellationTokenSource for timeout tracking in the every iteration, because it times out in every iteration. Under the hood, WaitAsync uses CancellationTokenSource.TryReset() in attempt to reuse the CTS instance. But CTS instance is in cancelled state in every iteration, because no one resumes the event. The disposed CTS is eligible for GC.

Memory profiler shows that the test produces 100 instances of the CTS. But all of them are in disposed state. If I uncomment the two lines of code with calls to GC, then the profile reports 0 survived instances of the CTS. It means that there is no memory leak.

[KostyaLocal]

Yes, it looks like everything is fine on your side. It’s just that adding the wait scheme via AsyncAutoResetEvent instead of polling caused a sharp increase in memory usage by almost +20% +25%, which is quite a lot. The dumps show a large number of created timers, but their total size is pretty small, so it doesn’t explain such growth. It’s possible that timers are being created very quickly and in large numbers; I’m still digging into it. Thanks for the quick response, Roman.

[KostyaLocal]

Hey, Roman, could you take a look at this?

Visual Studio’s profiler (Memory usage mode) shows that the managed heap size stays the same, but the process memory (private bytes) is slowly growing.

In this repro I’m using AsyncRunner instead of the production code, which normally calls wakeEvent.Set() directly.
Program.cs is below

using DotNext.Threading;
using System.Diagnostics;

await MemoryLeak();

static async Task MemoryLeak() {
    using AsyncAutoResetEvent wakeEvent = new(false);
    using var cts = new CancellationTokenSource();
    AsyncRunner.RunPeriodicAsync(() => wakeEvent.Set(), 10, cts.Token);
    const int delay = 10;
    var timer = Stopwatch.StartNew();
    for (var i = 0; i < 100_000; i++) {
        if (i % 2 == 0) wakeEvent.Set();
        var remaining = (double)Math.Max(0, delay - timer.ElapsedMilliseconds);
        await wakeEvent.WaitAsync(TimeSpan.FromMilliseconds(remaining), CancellationToken.None);
        timer.Restart();
    }

    GC.Collect();
    GC.WaitForPendingFinalizers();

    Debugger.Break();
}

return;


public static class AsyncRunner {
    public static void RunPeriodicAsync(Action func, int intervalMs, CancellationToken ct = default) {
        Task.Run(async () => {
            try {
                while (!ct.IsCancellationRequested) {
                    func();
                    await Task.Delay(intervalMs, ct);
                }
            }
            catch (Exception ex) {
                Console.WriteLine(ex.Message);
            }
        }, ct);
    }
}

[KostyaLocal]

IRL the symptoms are as follows: the displayed by top/ctop process memory is constantly growing, although at varying rates: from 0 to 5 MB/hour, as observed. I tried adding GC.Collect() once a minute - it didn't help. Although GCs activity is visible, the memory usage overall is steadily increasing hour by hour. Looks like unmanaged memory leak.

[KostyaLocal]

If I comment out the line with AsyncRunner, growth is also observed, but less active.

[sakno]

AsyncAutoResetEvent doesn't use the unmanaged memory allocation as well as everything in DotNext.Threading library. If the managed heaps remains unchanged, the root cause is somewhere else.

[sakno]

That's the picture for your code on my machine:

The unmanaged memory is growing at the startup, but in the end it remains stable.

[KostyaLocal]

AsyncAutoResetEvent doesn't use the unmanaged memory allocation as well as everything in DotNext.Threading library. If the managed heaps remains unchanged, the root cause is somewhere else.

While AsyncAutoResetEvent may not allocate explicit unmanaged memory, it does rely on internal primitives that can trigger unmanaged allocations indirectly. For example:

• Monitor.Enter/Exit uses OS synchronization objects under contention
• internal WaitNode, queues, continuations may allocate handles
• blocking paths can cause the CLR to allocate native WaitHandles behind the scenes
  and so on.

So even if DotNext.Threading doesn't allocate unmanaged memory directly, it can still be involved in producing patterns that lead to unmanaged growth or fragmentation, and the absence of visible managed heap growth does NOT guarantee the root cause lies elsewhere.