[Bug] helm installation breaks containerd config if [proxy_plugins] section already exist (and empty)

[botinca]

Problem Description

I'm trying to install nydus-snapshotter to exist k8s cluster as per article with command
helm install --wait --timeout 10m --dependency-update --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly
and config

  # -- Global Docker image registry.
  imageRegistry: "my.private.registry/ghcr-proxy"

name: nydus-snapshotter
pullPolicy: Always
hostNetwork: true
resources:
  requests:
    cpu: "0"
    memory: "0"
  limits:
    cpu: "1"
    memory: "1Gi"

dragonfly:
  enable: false #default true
  mirrorConfig:
  - host: "http://127.0.0.1:4001"
    auth_through: false
    headers:
      "X-Dragonfly-Registry": "https://ghcr.io"
    ping_url: "http://127.0.0.1:4003/healthy"

containerRuntime:
  initContainerImage: liubin/toml-cli:v0.0.7
  containerd:
    enable: true
    configFile: "/etc/containerd/config.toml"

After installation it brokes containerd config

containerd[2808038]: containerd: failed to load TOML: /etc/containerd/config.toml: (294, 2): duplicated tables

Expected Behavior

containerd should work after restart, config should be valid

Actual Behavior

an incorrect config (with duplicated section ) is generated

Containerd stops work containerd[2808038]: containerd: failed to load TOML: /etc/containerd/config.toml: (294, 2): duplicated tables

# grep -n proxy_plugins /etc/containerd/config.toml
264:[proxy_plugins]
294:[proxy_plugins]
295:  [proxy_plugins.nydus]

How to reproduce

1. As per article, install via helm
2. Login at any worker node and check containerd status

Environment Details

• Nydus-snapshotter version: containerd/nydus-snapshotter:v0.9.0
• Helm chart: chart=nydus-snapshotter-0.0.10 (current for today)
• toml image: liubin/toml-cli:v0.0.7
• Nydus version:
• Container runtime: containerd containerd.io 1.7.27 05044ec0a9a75232cad458027ca83437aae3f4da
• Operating System: Debian 11.11
• Kernel version: 6.1.0-0.deb11.21-amd64

Additional Information

kubectl describe daemonset nydus-snapshotter -n nydus-snapshotter

Name:           nydus-snapshotter
Namespace:      nydus-snapshotter
Selector:       app=nydus-snapshotter,component=nydus-snapshotter,release=nydus-snapshotter
Node-Selector:  <none>
Labels:         app=nydus-snapshotter
                app.kubernetes.io/managed-by=Helm
                chart=nydus-snapshotter-0.0.10
                component=nydus-snapshotter
                heritage=Helm
                release=nydus-snapshotter
Annotations:    deprecated.daemonset.template.generation: 1
                meta.helm.sh/release-name: nydus-snapshotter
                meta.helm.sh/release-namespace: nydus-snapshotter
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=nydus-snapshotter
                    component=nydus-snapshotter
                    release=nydus-snapshotter
  Service Account:  nydus-snapshotter-sa
  Init Containers:
   update-containerd:
    Image:      my.private.registry/ghcr-proxy/liubin/toml-cli:v0.0.7
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -cx
      etcContainerd=/etc/containerd/config.toml

      toml check $etcContainerd proxy_plugins.nydus
      if [ $? -eq 0 ]; then
        echo "nydus snapshotter has already configured."
        exit 0
      fi

      toml set --overwrite $etcContainerd plugins.\"io.containerd.grpc.v1.cri\".containerd.discard_unpacked_layers false
      toml set --overwrite $etcContainerd plugins.\"io.containerd.grpc.v1.cri\".containerd.disable_snapshot_annotations false
      toml set --overwrite $etcContainerd plugins.\"io.containerd.grpc.v1.cri\".containerd.snapshotter nydus

      **# toml command not support to set block, so just use cat command.
      cat << EOF >> $etcContainerd
      [proxy_plugins]
        [proxy_plugins.nydus]
          type = "snapshot"
          address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
      EOF**
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:        0
      memory:     0
    Environment:  <none>
    Mounts:
      /etc/containerd/config.toml from containerd-conf (rw)
  Containers:
   nydus-snapshotter:
    Image:      my.private.registry/ghcr-proxy/containerd/nydus-snapshotter:v0.9.0
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     0
      memory:  0
    Environment:
      ENABLE_NYDUS_OVERLAY:  false
    Mounts:
      /dev/fuse from fuse (rw)
      /etc/nydus/ from config (rw)
      /run/containerd-nydus from nydus-run (rw)
      /var/lib/containerd-nydus from nydus-lib (rw)
  Volumes:
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nydus-snapshotter
    Optional:  false
   nydus-run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/containerd-nydus
    HostPathType:  DirectoryOrCreate
   nydus-lib:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/containerd-nydus
    HostPathType:  DirectoryOrCreate
   fuse:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/fuse
    HostPathType:
   containerd-conf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/containerd/config.toml
    HostPathType:
  Node-Selectors:  <none>
  Tolerations:     <none>
Events:            <none>

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[imeoer]

Thanks! Could you help submit a PR to fix this? :)

[botinca]

I don't have completed PR (yet?), but I can share my proposal:

just amend block (with incorrect assumption that the toml don't know how to work with tables)

              cat << EOF >> $etcContainerd

              [proxy_plugins]
                [proxy_plugins.nydus]
                  type = "snapshot"
                  address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
              EOF

with just commands

toml set --overwrite $etcContainerd 'proxy_plugins.nydus.address' '/run/containerd-nydus/containerd-nydus-grpc.sock'
toml set --overwrite $etcContainerd 'proxy_plugins.nydus.type' 'snapshot'

It works, but produces not well-formated config

[proxy_plugins]

[proxy_plugins.nydus]

address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
type = "snapshot"