request help: Health Check not working in APISIX with standalone setup for load balacer functionality .

[taherbohri]

Issue description

Setup -

### ApisixRoute -

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: load-balancer-apisix-route
spec:
  ingressClassName: apisix
  http:
    - name: manageapi
      match:
        paths:
          - /hawkeye-management-service/apis/digital-bff
      upstreams:
      - name: service-v1-upstream
      - name: service-v2-upstream
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            headers:
              Authorization: "Basic ****"


### Upstreams -

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: service-v2-upstream
spec:
  scheme: https
  retries: 0
  passHost: node
  externalNodes:
  - type: Domain
    name: api-gateway.example.com
    weight: 1
    port: 443
  healthCheck:
    active:
      type: https
      httpPath: /v1/sadc/gn/healthcheck/status
      host: api-gateway.example.com
      port: 443
      healthy:
        httpCodes:
          - 200
          - 201
          - 202
        interval: "1s"      
        successes: 1    
      unhealthy:
        httpCodes:
          - 500
          - 503
          - 502
        interval: "1s"        
        httpFailures: 1
        tcpFailures: 1
        timeouts: 1
    passive:
      healthy:
        httpCodes:
          - 200
          - 201
          - 202
        successes: 1
      unhealthy:
        httpCodes:
          - 500
          - 503
          - 502
        httpFailures: 1
        tcpFailures: 1
        timeouts: 1



apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: service-v1-upstream
spec:
  scheme: https
  retries: 1
  passHost: node
  externalNodes:
    - type: Domain
      name: external-api-service.example.com
      weight: 1
      port: 443
  healthCheck:
    active:
      type: https
      httpPath: /hawkeye-management-service/health
      host: external-api-service.example.com
      port: 443
      healthy:
        httpCodes:
          - 200
          - 201
          - 202
        interval: "1s"       
        successes: 1      
      unhealthy:
        httpCodes:
          - 500
          - 503
          - 502
        interval: "1s"       
        httpFailures: 1
        tcpFailures: 1
        timeouts: 1
    passive:
      healthy:
        httpCodes:
          - 200
          - 201
          - 202
        successes: 1
      unhealthy:
        httpCodes:
          - 500
          - 503
          - 502
        httpFailures: 1
        tcpFailures: 1
        timeouts: 1

Even after setting up the heathcheck , where one of the health check call will return 503 , the calls are still routing between the 2 upstreams , even when of the upstream is down and heathcheck returns 503.

After checking the apisix docs for healthcheck I got this -

That the health check is deprecated for the standalone deployment .

Is there any solution for this ?

Environment

Environment

• APISIX version: 3.13.0
• APISIX Ingress Controller: 2.0.0-rc2
• Deployment mode: Standalone (apisix-standalone provider)
• Gateway API: v1
• Platform: Kubernetes

[bzp2010]

@taherbohri

In standalone deployments (not only standalone deployment, but also all ingress controller scenarios), health checks should actually be performed by Kubernetes probes and exposed to applications through Kubernetes services and endpoints.
The Ingress Controller is responsible for tracking the endpoint/endpointslice associated with every active Kubernetes service, where the corresponding pod IP is stored. When a pod is not in a ready state, it will not appear in this list until the probe informs Kubernetes that it is ready.
In most cases, APISIX neither needs nor should use its built-in health check capabilities (when using Ingress Controller). Instead, Kubernetes' native capabilities should be utilized first. Only when these prove completely inadequate should APISIX's options be considered.

For the reasons stated above, I have decided that the underlying technology of the Ingress Controller (ADC) does not support this option. Currently, this is an artificial limitation that may be lifted in the future based on actual requirements.
When I developed this feature, there was no compelling reason to add support for them, even though doing so would have required only a few dozen lines of code and some additional testing.
ref: https://github.com/api7/adc/blob/main/libs/backend-apisix-standalone/README.md#differences-in-upstream

If you have more compelling reasons, you may present them to persuade me to reconsider this decision.

[taherbohri]

@bzp2010
The usecase for us is that we need to call a service which is outside of Kubernetes cluster using "ExternalName" service. For these services there is no way for Kubernetes do perform healthchecks. So we wanted to use APISIX upstream healthcheck capabilities for this.

[bzp2010]

@taherbohri This does appear to be a reasonable use case.

I will consider adding it to the roadmap for future implementation.

[bzp2010]

This has been implemented by api7/adc#333 (dependency of this project) and is expected to be available in the next apisix-ingress-controller release.

[bzp2010]

Done by #2574, it will be included in the next release, rc5.