calico fails to create ip route while wep is present for the pod

[ajayudayagiri-hpe]

Calico falied to create ip route entry for few pods which has an wep entry. This happens randomly on different cluster/node when rebooted.

Expected Behavior

Calico should recreate all ip routes for pods post reboot.

Current Behavior

Post reboot on clusters randomly few pods are stuck in CrashLoopBackOff state due to network rechability issue. This is due to absence of ip route for the pod ip and cali* interface.

Pod has IP assigned

core@cop-node-88-65:~$ lspodwide | grep pvos-switch-state-processor-1
gravity            pvos-switch-state-processor-1                                                                                    0/2     CrashLoopBackOff        1073 (2m24s ago)   13d     172.16.111.99    cop-node-88-65.arubacorp.net   <none>           <none>

Calicoctl WEP list shows the IP and Interface

core@cop-node-88-65:~$ calicoctl get wep -n gravity | grep pvos-switch-state-processor-1
gravity     pvos-switch-state-processor-1                                                           cop-node-88-65.arubacorp.net   172.16.111.99/32    cali8536608cf3f

Ifconfig output

core@cop-node-88-65:~$ ifconfig cali8536608cf3f
cali8536608cf3f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ping output

core@cop-node-88-65:~$ ping 172.16.111.99
ping: connect: Invalid argument
core@cop-node-88-65:~$ ping 172.16.111.99
ping: connect: Invalid argument

ip route output

core@cop-node-88-65:~$ ip route | grep 172.16.111.99
core@cop-node-88-65:~$
core@cop-node-88-65:~$ ip route | grep cali8536608cf3f
core@cop-node-88-65:~$

No iptables refresh interval is modified, everything is default.

2025-10-09 15:24:32.953 [INFO][81] felix/int_dataplane.go 2180: Will refresh IP sets on timer interval=1m30s
2025-10-09 15:24:32.953 [INFO][81] felix/int_dataplane.go 2180: Will refresh routes on timer interval=1m30s
2025-10-09 15:24:32.953 [INFO][81] felix/int_dataplane.go 2180: Will refresh XDP state on timer interval=1m30s

Possible Solution

Felix should be able to reconcile with pod interfaces in WEP and create missing route. However, it is not recreating even after 12 hours of bootup.

Steps to Reproduce (for bugs)

1. Create kubernetes cluster with calico - 3/5/7 nodes
2. Bring up enough pods - 300-400 pods/node
3. Reboot cluster
4. After few minutes find few pods in CrashLoopBackOff with route missing

One observation is that if calico-node pod is restarted, this route is reconciled with WEP and entry is created in ip route table, thus recovering the pod state to Running.

Context

Cluster is unstable after reboot as pods are never recovered when stuck in this state. Interested to know if any specific parameter can be tuned to fix this issue.

Your Environment

• Calico version: v3.30.0
• Calico dataplane (bpf, nftables, iptables, windows etc.): iptables
• Orchestrator version (e.g. kubernetes, openshift, etc.): Kubernetes v1.32.5
• Operating System and version: Ubuntu 22.04.5 LTS

[caseydavenport]

Are you using Calico in etcd mode or in kdd mode? Did you install with the tigera/operator or via calico.yaml or something else?

In general, Felix should and can program missing routes - either it's hitting an error, or it is unaware of the pod for some reason.

Rebooting nodes can cause issues - especially I have seen problems with container runtimes failing to properly tell Calico about new / deleted containers. Not sure if that is what is happening here, though.

Do you have a full Felix log we could look at?

[ajayudayagiri-hpe]

@caseydavenport - Here are further details -
Calico is in kdd mode
installed through calico.yaml
Older Felix logs are rotated. However, I have cluster in same state now with few pods missing ip route, there is no error log I see in calico pods.

Also, one more observation I made while trying to understand if routeRefreshInterval of felix is actually working - I have deleted ip route of one of the existing/running pod. Felix was successfully able to add it back in 90seconds (routeRefresh). However, on the same node/cluster, the route missing for the pod during the reboot is still not added back.

Is felix maintaining a state here where it somehow missed few pods during reboot and only performs refresh on successful pod list ?

[ajayudayagiri-hpe]

@caseydavenport - just checking if you got a chance to go through my comment.

[caseydavenport]

@ajayudayagiri-hpe yep, I have - I haven't had a chance to attempt a reproduction yet. Barring that, I would still need a full felix log from one of the rebooted nodes to understand what is happening here a bit better.

[ajayudayagiri-hpe]

@caseydavenport - Ok, I will get those logs meanwhile.

[ajayudayagiri-hpe]

@caseydavenport - I have reproduced the issue again and captured calico-node logs in this github link.

The steps to reproduce were same again, a reboot of cluster will land some random pods into this scenario.

core@cop-node-88-63:~$ lspodwide | grep 172.23.22.107
ingress-nginx      nginx-device-ingress-ds-574h9                                                                                    0/1     CrashLoopBackOff         11 (31s ago)      8d      172.23.22.107    cop-node-88-63.arubacorp.net   <none>           <none>

core@cop-node-88-63:~$ ping 172.23.22.107
ping: connect: Invalid argument

core@cop-node-88-63:~$ ip route | grep 172.23.22.107

core@cop-node-88-63:~$ calicoctl get wep -A | grep 172.23.22.107
ingress-nginx      nginx-device-ingress-ds-574h9                                                                                    cop-node-88-63.arubacorp.net   172.23.22.107/32    caliae0611e0230

core@cop-node-88-63:~$ ip route | grep caliae0611e0230

core@cop-node-88-63:~$ ifconfig caliae0611e0230
caliae0611e0230: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

core@cop-node-88-63:~$ uptime
 16:35:29 up 15 min,  2 users,  load average: 23.08, 54.04, 33.18

As shown above, there is no ip route created for this pod with ip 172.23.22.107. However, calico wep is still present. The log file contains logs from all calico-node instances. As this issue is random, I have collected logs from all pods.

Please go through and let us know if any further information is required.

[ajayudayagiri-hpe]

@caseydavenport - Did you get a chance to look into the logs I shared ?

[caseydavenport]

Yep, I've taken a first pass at them but still digging in.

calico-node-z8r4x calico-node 2025-10-23 16:32:27.916 [INFO][77] felix/int_dataplane.go 2201: Received *proto.WorkloadEndpointUpdate update from calculation graph. msg=id:{orchestrator_id:"k8s" workload_id:"ingress-nginx/nginx-device-ingress-ds-574h9" endpoint_id:"eth0"} endpoint:{state:"active" name:"caliae0611e0230" profile_ids:"kns.ingress-nginx" profile_ids:"ksa.ingress-nginx.default" ipv4_nets:"172.23.22.107/32"}

Calico does seem to be receiving the workload endpoint.

Although immediately after it seems to identify the interface corresponding to that WEP as being DOWN:

calico-node-z8r4x calico-node 2025-10-23 16:32:27.918 [INFO][77] felix/endpoint_mgr.go 1505: Skipping configuration of interface because it is oper down. ifaceName="caliae0611e0230"

Which doesn't seem to match your ifconfig output above, but seems relevant. The logs seems to show that the interface state is toggling a fair bit (snippet below is filtered to logs referencing the interface)

calico-node-z8r4x calico-node 2025-10-23 16:27:11.907 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=514 ifaceName="caliae0611e0230" state="down"
calico-node-z8r4x calico-node 2025-10-23 16:27:11.907 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=514 ifaceName="caliae0611e0230" state="up"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=501 ifaceName="caliae0611e0230" state="down"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1634: Linux interface addrs changed. addrs=set.Set{} ifaceName="caliae0611e0230"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=501 ifaceName="caliae0611e0230" state="up"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=501 ifaceName="caliae0611e0230" state="down"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1590: Linux interface state changed. ifIndex=501 ifaceName="caliae0611e0230" state=""
calico-node-z8r4x calico-node 2025-10-23 16:27:12.089 [INFO][77] felix/int_dataplane.go 1634: Linux interface addrs changed. addrs=<nil> ifaceName="caliae0611e0230"
calico-node-z8r4x calico-node 2025-10-23 16:27:12.178 [INFO][77] felix/int_dataplane.go 2259: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"caliae0611e0230", State:"down", Index:501}
calico-node-z8r4x calico-node 2025-10-23 16:27:12.179 [INFO][77] felix/int_dataplane.go 2286: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"caliae0611e0230", Addrs:set.Typed[string]{}}
calico-node-z8r4x calico-node 2025-10-23 16:27:12.179 [INFO][77] felix/hostip_mgr.go 84: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"caliae0611e0230", Addrs:set.Typed[string]{}}
calico-node-z8r4x calico-node 2025-10-23 16:27:12.179 [INFO][77] felix/int_dataplane.go 2259: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"caliae0611e0230", State:"up", Index:501}
calico-node-z8r4x calico-node 2025-10-23 16:27:12.179 [INFO][77] felix/int_dataplane.go 2259: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"caliae0611e0230", State:"down", Index:501}
calico-node-z8r4x calico-node 2025-10-23 16:27:12.179 [INFO][77] felix/int_dataplane.go 2259: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"caliae0611e0230", State:"", Index:501}

[ajayudayagiri-hpe]

Ok. Also, i wonder why calico is not adding back the missing route during route refresh.

[caseydavenport]

It won't add the route back if it thinks the interface is down.