all: netbsd-arm--rpi3: all Go (not subrepo) builds time out

[bsiegert]

The netbsd-arm--rpi3 builder has not had a successful "gotip" (main Go tree) build in weeks. All the builds time out after 3 to 4 hours. What is puzzling to me is that the build never seems to start. Example:

https://luci-milo.appspot.com/ui/p/golang/builders/ci/gotip-netbsd-arm/b8699782187822189569/overview

The full output in there is:

[I2025-10-28T08:31:48.350764Z 28706 0 sink.go:354] SinkServer: warm-up started
[I2025-10-28T08:31:48.352359Z 28706 0 sink.go:427] SinkServer: starting HTTP server...
[I2025-10-28T08:31:48.359233Z 28706 0 sink.go:359] SinkServer: warm-up ended
[I2025-10-28T08:31:48.360776Z 28706 0 cmd_stream.go:577] rdb-stream: starting the test command - ["/home/swarming/.swarming/w/ir/cache/tools/bin/result_adapter" "go" "-v=false" "-dump-json" "/home/swarming/.swarming/w/ir/x/w/dist.testjson" "--" "/home/swarming/.swarming/w/ir/x/w/goroot/bin/go" "tool" "dist" "test" "-json"]

I would expect some test output after this, but there is nothing.

The other machine can do successful builds, e.g. https://luci-milo.appspot.com/ui/p/golang/builders/ci/gotip-netbsd-arm/b8699866658261624945/overview.

I haven't changed anything in the hardware or software lately. Any tips to debug?

[mknyszek]

Hm... I think we need to figure out which command exactly is hanging. Assuming these builders are hooked up with gomote, gomote does set up an identical environment for running tests, so that might be worth trying, specifically to rule out it being go tool dist test -json that's hanging. If we know that's not hanging, then that suggests one of the LUCI tools is broken for some reason. Unfortunately we don't have an easy way to target a specific machine for a gomote, but this is in theory very feasible. Perhaps we should prioritize that.

One thing to possibly try is to maybe clear out the CIPD caches on the machine?