Talosctl using wrong addresses for healthcheck

[golebiewsky]

Hello

I'm running a cluster where worker nodes are on tailscale and control plane ones other are not.
Everything seem to be working fine in the cluster, but when doing e.g. talosctl health it tries to use tailscale IPs for some of the nodes, but since Talos doesn't actually use that interface for API - it fails, e.g.

talosctl health
discovered nodes: ["TAILSCALE_IP_A" "PUBLIC_IP_C" "TAILSCALE_IP_B" "PUBLIC_IP_D" "PUBLIC_IP_E" "PUBLIC_IP_F"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: rpc error: code = DeadlineExceeded desc = context deadline exceeded
waiting for apid to be ready: 2 errors occurred:
	* TAILSCALE_IP_A: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp TAILSCALE_IP_A:50000: i/o timeout"
	* TAILSCALE_IP_B: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp TAILSCALE_IP_B:50000: i/o timeout"
...

Whats interesting that it doesn't do that for one node, just by chance tailscale IP is listed as second (probably cause Public IP has lower number in first octet) and with it uses correct - public IP:

talosctl get affiliates -n 141.98.216.153
NODE             NAMESPACE   TYPE        ID     VERSION   HOSTNAME     MACHINE TYPE   ADDRESSES
...
PUBLIC_IP_B      cluster     Affiliate   yyy    5         w987         worker         ["TAILSCALE_IP_B","PUBLIC_IP_B"]
PUBLIC_IP_C      cluster     Affiliate   xxx    1         w123         worker         ["PUBLIC_IP_C","TAILSCALE_IP_C"]

Maybe the solution would be to just remove tailscale address from discovery..? Please advise.

[smira]

talosctl has no way to know which address it needs to use, it has explicit flags for you to pass expected node addresses.

[golebiewsky]

But health lets me use only one node address - doesn't it?

[smira]

Yes, send the IPs with are not Tailscale IPs via flags.

If it doesn't work, open a specific issue with command your run, output, support bundle.

[golebiewsky]

Yup, thats what I did, then talos health throws this waiting for all k8s nodes to report: can't find expected node with IPs ["PUBLIC_IP"]
I wil open a ticket then, thanks.

[smira]

See above, if you register Kubernetes with a different IP than the one you specify to talosctl health, that is expected.

It's your setup/choice, so generic health can't run all checks. You can run your checks yourself.

[golebiewsky]

It wasn't 'my choice' though, seems just like adding tailscale at different moments cause that situation. I will test with another cluster.

[smira]

it was your choice in a sense that you run multi-homed cluster and you didn't care to specify which address you want.

https://www.talos.dev/v1.9/introduction/prodnotes/#multihoming