Skip to content

Unable to install Quickwit with IPv6 network #118

@erebe

Description

@erebe

Hello,

I am reporting an issue where I can't install Quickwit chart when the kubernetes cluster is in IPv6.
The reason is due to the healthchecks port, not listening for IPv6.

My value file.

❯ cat quickwit.yaml
environment:
  # QW_LISTEN_ADDRESS: "::"
  QW_METASTORE_URI: s3://quickwit/quickwit-indexes

config:
  # listen_address: "::"
  default_index_root_uri: s3://quickwit/quickwit-indexes
  storage:
    s3:
      flavor: minio
      endpoint: http://[fd00:cafe::6]:9001
      region: eu-east-1
      access_key_id: admin
      secret_access_key: xxxx

tolerations:
  - key: "kubernetes.io/hostname"
    operator: "Equal"
    value: "toybox"
    effect: "NoSchedule"

The helm command used to install quickwit

helm install quickwit quickwit/quickwit -f quickwit.yaml --wait --atomic

The result
image

All pods fails on the startup probe

Startup probe failed: Get "http://[fd01::430]:7280/health/livez": dial tcp [fd01::430]:7280: connect: connection refused

When I get into the machine to check the openned port (I use nsenter to get access to the network namespace)

erebe@toybox ~ [SIGINT]> sudo nsenter -n -t (ps aux | grep 'metastore' | grep -v grep | awk '{print $2}') -- ss -lntp
State           Recv-Q          Send-Q                   Local Address:Port                   Peer Address:Port          Process
LISTEN          0               128                            0.0.0.0:7280                        0.0.0.0:*              users:(("quickwit",pid=200495,fd=13))
LISTEN          0               128                            0.0.0.0:7281                        0.0.0.0:*              users:(("quickwit",pid=200495,fd=12))

We see that Quickwit is listening on 0.0.0.0 but not on [::] for IPv6.
We can confirm with curl

# IPv4 works
erebe@toybox ~ [1]> sudo nsenter -n -t (ps aux | grep 'metastore' | grep -v grep | awk '{print $2}') -- curl http://10.42.4.34:7280/health/livez
true⏎  

#IPv6 hangs
sudo nsenter -n -t (ps aux | grep 'metastore' | grep -v grep | awk '{print $2}') -- curl http://[fd01::430]:7280/health/livez
xxxx

Changing the listening address to :: in the config makes everything works and should be working also for both IPv4 and IPv6 networks

erebe@toybox ~ [SIGINT]> sudo nsenter -n -t (ps aux | grep 'metastore' | grep -v grep | awk '{print $2}') -- ss -lntp
State           Recv-Q          Send-Q                   Local Address:Port                   Peer Address:Port          Process
LISTEN          0               128                                  *:7280                              *:*              users:(("quickwit",pid=204513,fd=13))
LISTEN          0               128                                  *:7281                              *:*              users:(("quickwit",pid=204513,fd=12))

Expected result

The chart should use by default for listening address :: instead of 0.0.0.0
One strange thing to note, I don't know if it is linked to the base image of the container, but usually 0.0.0.0 bind also on IPv6 in order to ease transition from IPv4 to IPv6.
In the case of Quickwit, it does not happen and only bind for IPv4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions