DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled. - ADO self hosted build agent

[GeorgeB-RIB]

I'm stuck. Can anybody assist me? We are running a Azure DevOps self hosted linux agent in a docker container and auto-scaling it using KEDA in our AKS cluster.
I'm trying to use dotnet-testcontainers to run unit and integration tests using the following CI pipeline and docker-compose file:
CI Pipeline:

 parameters:
  - name: buildConfiguration
    displayName: Build Configuration
    type: string
    default: Release
  - name: agentPool
    displayName: Build Agent
    type: string
    default: keda-ubuntu-testcontainer-devops-agents
  - name: repositoryName
    type: string
    default: TestContainers-POC
  - name: repositoryResourceName
    type: string
    default: repoBIM6Dpoc
  - name: slnBuildPath
    type: string
    default: 'TestContainerSeleniumTest/TestContainerSeleniumTest.sln'

resources:
  repositories:
    - repository: repoTcPoc
      type: git
      name: TcPocProj/TC POC
      ref: us/selenium_testcontainers_poc

trigger: none

jobs:
  - job: RUN_TEST_CONTAINER_SELENIUM
    pool: ${{ parameters.agentPool }}
      # vmImage: ubuntu-latest
    continueOnError: false
    workspace:
      clean: outputs
    steps:
      - checkout: ${{parameters.repositoryResourceName}}
        clean: true
        persistCredentials: true
        path: ${{ parameters.repositoryName }}
        displayName: Checkout TestContainers POC Repo
      
      - task: DotNetCoreCLI@2
        inputs:
          command: 'build'
          projects: '$(Agent.BuildDirectory)/${{ parameters.repositoryName }}/${{ parameters.slnBuildPath }}'
        displayName: 'Build Solution'

      - task: DotNetCoreCLI@2
        inputs:
          command: 'test'
          projects: '$(Agent.BuildDirectory)/${{ parameters.repositoryName }}/${{ parameters.slnBuildPath }}'
          arguments: '--logger:trx'
        displayName: 'Test Solution'

docker-compose file:

version: '3.8'
services:
  test-azp-agent-ubuntu:    
    container_name: test-azp-agent-ubuntu    
    image: 'test-azp-agent-ubuntu-repository:1.0'
    environment:
      AZP_URL: 'https://myorg.visualstudio.com'
      AZP_TOKEN: '<ADO-PAT>'
      AZP_POOL: 'keda-ubuntu-testcontainer-devops-agents'
      AZP_STARTUP: '--once'
      AZP_AGENTDIR: 'agent'
      AZP_AGENT_NAME: 'DO-NOT-REMOVE'
      DOCKER_HOST: 'unix:///var/run/docker.sock'  # Ensure Docker endpoint is set
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The ADO agent docker image has docker installed and running along with all of the other build agent capabilities that we require.

I keep running into the following error (see output) when attempting to run testcontainers on our hosted linux agent. (It works perfectly using MS-Hosted runner-agents that run on vmImage as opposed to our docker image.
Testscontainers Error build output:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed Test2 [1 m 2 s]
  Error Message:
   DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled.
  Stack Trace:
     at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 236
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, ILogger logger, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 248
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, ILogger logger, Boolean isWindowsEngineEnabled, CancellationToken ct) in /_/src/Testcontainers/Containers/ResourceReaper.cs:line 135
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /_/src/Testcontainers/Clients/TestcontainersClient.cs:line 297
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 416
   at Testcontainers.WebDriver.WebDriverContainer.UnsafeCreateAsync(CancellationToken ct) in /_/src/Testcontainers.WebDriver/WebDriverContainer.cs:line 74
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /_/src/Testcontainers/Containers/DockerContainer.cs:line 280
   at TestContainerSeleniumTest.Tests2.Test2() in /azp/_work/1/BIM6D-POC/TestContainerSeleniumTest/TestContainerSeleniumTest/UnitTests2.cs:line 22
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
   at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()

  Standard Output Messages:

It's obvious that RYUK cannot start up but I cannot figure out why. Is it due to a misconfiguration or because the type of DnD scenario I'm trying to make work?

[HofmeisterAn]

Can you please try to mount the raw socket:

/var/run/docker.sock.raw:/var/run/docker.sock

[GeorgeB-RIB]

Thanks for your reply. After trying that configuration in the volume mount I still get the same error.
From the stack trace it would appear as if the fails to get the container state maybe due to failing to create it in the first place.
Is there any way I can get more detailed logs to investigate if the testcontainer is able to be created using docker.service and dockerd?
I have pulled the testcontainers repo from github to investigate this error further.

[HofmeisterAn]

It looks like Testcontainers cannot establish a connection with the Ryuk container (the host running the tests tries to connect to the container). In environments like yours, this can sometimes be difficult to set up. You may need to set the TESTCONTAINERS_HOST_OVERRIDE environment variable to your container host. Can you please try it manually first?

Start Ryuk, then attempt to connect:

docker run -e RYUK_CONNECTION_TIMEOUT=120s -v /var/run/docker.sock:/var/run/docker.sock -p 8080:8080 testcontainers/ryuk:0.9.0
# I'm not sure if localhost is correct for your environment.
nc -N localhost 8080

[GeorgeB-RIB]

Hi @HofmeisterAn. Firstly: If I manually create the RYUK container using the docker host do I need to disable it in the test classes running different modules? (i.e. Dotnet.Testcontainers.PostgreSql, DotNet.Testcontainers.Builders)
Secondly: Is there a way I can diagnose the RYUK issue by increasing the verbosity of the Dotnet.TestContainers logs and uotput them to the build agent logs?
I have tried using the LoggerFactory in my test classes without much success:

var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.AddConsole();
    builder.SetMinimumLevel(LogLevel.Debug); // Set log level to Debug
});
_logger = loggerFactory.CreateLogger<CustomerServiceTest>();

// Initialize Docker client
_dockerClient = new DockerClientConfiguration(new Uri("unix:///var/run/docker.sock")).CreateClient();

// Create a custom network with additional options
_network = new NetworkBuilder()
    .WithName("my_custom_network")
    .WithDriver(NetworkDriver.Bridge)
    .WithOption("com.docker.network.bridge.enable_icc", "true")
    .WithOption("com.docker.network.bridge.enable_ip_masquerade", "true")
    .Build();

// Initialize Ryuk container
_ryuk = new ContainerBuilder()
    .WithImage("testcontainers/ryuk:0.9.0")
    .WithNetwork(_network)
    .WithNetworkAliases("ryuk")
    .WithEnvironment("TESTCONTAINERS_RYUK_CONTAINER_TIMEOUT", "90")
    .WithEnvironment("TESTCONTAINERS_RYUK_DEBUG", "true")
    .WithPrivileged(true)
    .Build();

[HofmeisterAn]

You should not run Ryuk that way. What I meant was to run it manually using the CLI and test if it runs successfully. Then, try connecting using Netcat or a similar tool to ensure your environment is properly configured.