bug: Upstream/Route state not refreshed after pod restart → 502s (adc v0.21.1, AIC 2.0.0-rc4) - helm-chart 2.11.6 #2589
Description
Current Behavior
Current Behavior
Summary
After a workload pod restarts and receives a new IP, APISIX continues routing to the old endpoint. In addition, route-level custom plugins (auth/credentials) are not refreshed. This leads to 502 Bad Gateway responses until a manual resync.
Also, when a new route is created via CDR, it does not appear in the admin route list (returns 404), even though the route creation itself succeeds. Deleting and recreating the route does not resolve the issue.
Environment
- helm-chart: 2.11.6
- adc: 0.21.1 (custom image for linux/arm64)
- apisix-ingress-controller: 2.0.0-rc4
- apisix: 3.13
- Traffic config: APISIX CRDs (ApisixRoute, etc.)
2025-09-29T10:17:21Z Service restarted
New pod IP: 10.13.195.107
APISIX admin view (after restart) still shows old node:
"upstream": {
"nodes":[{"host":"10.13.197.151","port":3000,"weight":100,"priority":0}]
}
APISIX then attempts to route to a non-existent pod → 502 Bad Gateway.
Expected
- New routes created via CRDs should appear in admin and work as expected.
- Upstream nodes should update promptly to reflect the new pod IP(s).
- Route custom plugins (auth/credentials) should reconcile with the current CRD state.
Actual
- New routes are missing in admin (
https://127.0.0.1:9180/apisix/admin/routes) → 404 Not Found. - Upstream continues pointing to the previous pod IP.
- Route custom plugins are not refreshed.
- Requests return 502 until a manual resync.
Notes
- We build our own
adc:0.21.1image for arm64 (binary pulled from the linked release asset).
Versions
- APISIX: 3.13
- Helm-chart: 2.11.6
- Ingress-controller: 2.0.0-rc4 (arm64)
- ADC: 0.21.1 (arm64)
- OS: (output of
uname -a)
Related Links
- PR: fix(apisix-standalone): conflict between upstream nodes and service discovery api7/adc#344
- Release: https://github.com/api7/adc/releases/tag/v0.21.1
Expected Behavior
No response
Error Logs
No response
Steps to Reproduce
Steps to Reproduce
502 Bad Gateway.
- Deploy a Service/Deployment behind an ApisixRoute.
- Confirm upstream node(s) in APISIX match the running pod IP(s).
- Restart the pod (e.g., kubectl rollout restart deploy/).
- Observe new pod IP via kubectl get pod -o wide.
- Query APISIX admin or /status to inspect upstream nodes.
- Send traffic to the route.
404 Not Found
- Create a new route
- Confirm that the route is created successfully
`status:
conditions:- message: Sync Successfully
observedGeneration: 1
reason: ResourcesSynced
status: "True"
type: ResourcesAvailable`
- message: Sync Successfully
- Observe route via kubectl get apisixroute -A
- Route is not appearing in APISIX admin route
- 404 when try to reach the new route.
Environment
- APISIX Ingress controller version (run
apisix-ingress-controller version --long) - Kubernetes cluster version (run
kubectl version) - OS version if running APISIX Ingress controller in a bare-metal environment (run
uname -a)