Workflow runs stuck in queued state for far too long

[yaxiongzhao-genesis]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install action runner controller with helm:

[27/10/25 2:10:52] ➜  playground git:(main) cat actions-runner-controler/values-arc.yml 
githubConfigSecret: arc-github-pat-secret%
helm upgrade --install arc \
    --values actions-runner-controler/values-arc.yml \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

2. Install runner scale set using helm:

[27/10/25 3:05:00] ➜  playground git:(main) cat actions-runner-controler/values-runner-scale-set.yml 

githubConfigSecret: arc-github-pat-secret
githubConfigUrl: https://github.com/yaxiongzhao-genesis/gs-core
runnerScaleSetName: gpu-runners

# Optional: Configure minimum and maximum runners
minRunners: 1
# maxRunners: 10

# Optional: Configure runner timeout
runnerTimeout: 10800 # 3 hours

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        # GPU resource requests and limits
        resources:
          requests:
            nvidia.com/gpu: 1
          limits:
            nvidia.com/gpu: 1

helm diff upgrade --install gpu-runners \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--values actions-runner-controler/values-runner-scale-set.yml

3. Launch a demo workflow by manually dispatching the workflow on github:
 action.yml
[27/10/25 3:07:32] ➜  gs-core git:(main) cat .github/workflows/demo-self-hosted-runner.yml 
name: Demo - self-hosted runner

# Trigger the workflow on push to main branch or manual trigger
on:
  workflow_dispatch:
  push:
    branches:
      - main
jobs:
  hello-world:
    # Run on CoreWeave ARC runner with GPU support
    runs-on: gpu-runners

    steps:
      - uses: actions/checkout@v4
      - name: Print system information
        run: |
          echo "🚀 Hello World from CoreWeave!"
          echo "================================"
          echo "Runner: $(hostname)"
          echo "User: $(whoami)"
          echo "Date: $(date)"
          echo "PWD: $(pwd)"
          echo "OS: $(uname -a)"

      - name: Check GPU availability
        run: |
          echo "🔍 Checking GPU availability..."
          if command -v nvidia-smi &> /dev/null; then
            echo "✅ nvidia-smi found - GPU support available!"
            nvidia-smi
          else
            echo "❌ nvidia-smi not found - checking for GPU devices..."
            ls -la /dev/nvidia* 2>/dev/null || echo "No NVIDIA devices found"
          fi

4. Watch the job to be picked up, and it hasn't been picked up for up to 30 minutes

Describe the bug

The workflow runs not picked up by action runners

Describe the expected behavior

The action runners pick up the workflow runs

arc-gha-rs-controller-6865f855c9-vhjhx-logs.txt

gpu-runners-7c4844cc-listener-logs.txt

Additional Context

N/A

Controller Logs

https://gist.github.com/yaxiongzhao-genesis/8cb38e74945d99c48c6c468e5381629b

Runner Pod Logs

https://gist.github.com/yaxiongzhao-genesis/2acff0fb2404faebc40d350b6213bb06

https://gist.github.com/yaxiongzhao-genesis/bb77ddf0fdee1d206c81362cadba59d7

[decimusphostle]

I was having this very issue for the longest time. I don't have a good solution yet, so I will be following this space to see what the 'official response' is. Having said that here are a couple of troubleshooting 'notions'/questions:

i) In the respective namespaces I presume you are seeing the controller and listener pods. But do you see the runner pods running? (In my case with 0.13.0, I was constantly seeing the runner pods failing in a loop - also EXTREMELY FRUSTRATINGLY: without any logs and then they would not be scheduled after the 'retries' were exhausted.)

ii) Are you seeing 'Runner scale sets' and the 'Available runners' being registered with your repo's 'Runners' (on the https://github.com/path/to/repo/actions/runners?tab=self-hosted side)? They both need to be there i.e. 'Runner scale sets' and 'Available runners'.

iii) And lastly, have you tried dind for your runner scale set 'install'? (FWIW, that seemed to get the 'runners' from flapping for me and coupled with 0.12.1 it seemed to work.) i.e. set this in your 'runner scale set'-helm chart values:

containerMode:
  type: "dind"

I spent way too much time trying to resolve this to the point of giving up. Not sure how or why it started working - for me - after these changes?! 🤷‍♂️

Will wait to see what the 'official response' is - as everything I mentioned above is as a n00b.

[nikola-jokic]

Hey @yaxiongzhao-genesis,

From the log, you can see that the controller fails to fetch the secret containing credentials which it needs in order to authenticate. After a while, I'm not sure if you fixed the secret or what happened, but as soon as the controller was able to fetch the secret, it continued working. Please check on your cluster if something delays fetching the secret. This is unlikely ARC issue. Unless we are able to fetch the secret in the proper format (which I can see that you specified the secret name with githubConfigSecret: arc-github-pat-secret.

The log line is:

2025-10-24T22:37:47Z	ERROR	EphemeralRunnerSet	Failed to clean up EphemeralRunners	{"version": "0.13.0", "ephemeralrunnerset": {"name":"gpu-runners-4w4z7","namespace":"yaxiong-dev"}, "error": "failed to resolve app config: failed to get kubernetes secret: \"yaxiong-dev/arc-github-pat-secret\""}

[nikola-jokic]

Hey @decimusphostle,

You need to understand why are your runners failing. The controller itself doesn't modify the spec core spec of your runner. You can see the pod yaml definition, and try to understand what caused the backoffs.
The way you should think about the controller is that the controller spins up the pod according to your template, applying few environment variables and requred fields for the container named runner.
You are in complete control of the spec.
The controller only reacts the termination state of the ephemeral runner pod. If the pod fails, and it did not start the workflow, we will retry it with backoff. Ideally, backoffs should be done reached at all (but obviously, failures can occur so we need to retry). So, if you see frequent backoffs, you will need to look over the root cause. It might be due to a runner pod spec, or it might be something external killing the pod.

[decimusphostle]

Hello @nikola-jokic ,

Thanks for the response.

You need to understand why are your runners failing.

I would love to do that - sadly I have not been able to figure out a way to do that.

You can see the pod yaml definition...

Any suggestions on specific things to look for? FWIW, the secret, the repo et al are all seemingly configured correctly.

You are in complete control of the spec.

The spec for the runner is the 'off-the-shelf' option i.e. nothing has been changed other than passing in values that specify properties like repo, secret, etc.

So, if you see frequent backoffs, you will need to look over the root cause. It might be due to a runner pod spec, or it might be something external killing the pod.

And the root cause analysis on runner pods that get killed off in quick succession and go away after the backoff kicks in has been tricky. describe does not show anything obvious and logs for the runners that vanish that quickly seems to be hard to get or non-existent (at least as far as my bash hacks and/or k9s can discern). As mentioned earlier the runner pod spec is unchanged - as far as I can tell. So not sure what I could do in this situation.