Skip to content

Reconciliation requiring StatefulSet recreation results in new PVC, with old data orphaned #1151

@tcbennun

Description

@tcbennun

Let's say I use this YAML to spec my EMQX resource.

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx
  namespace: emqx
spec:
  image: emqx/emqx:5.9.0
  coreTemplate:
    spec:
      resources:
        requests:
          cpu: 250m
          memory: 1Gi
      volumeClaimTemplates:
        storageClassName: standard
        resources:
          requests:
            storage: 1Gi
        accessModes:
          - ReadWriteOnce
      replicas: 1

One day, I realise I'm throttling out on CPU. I change cpu: 250m to cpu: 500m, and apply the new config (in my case I'm using Terraform, FWIW).

My EMQX Operator reconciles, and realises it needs to create a new StatefulSet. In doing so, it also creates a brand-new persistent volume claim. At the end of the day, my EMQX cluster is running anew, with none of its persisted data.

I can certainly manually copy the data from the old, orphaned PVC to the new one, but that kind of defies the point, does it not?

I observed this on operator 2.2.29.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions