Changing config doesn't lead to recreation by emqx operator

[lukas-mertens]

Describe the bug
I did a change to my emqx config like this:

for my emqx resource.

I also changed mounts in my resource:

Now, after performing a sync using argocd, the emqx correctly changed the mount. The configmap which is managed by the emqx-operator was not recreated however. This lead to the new pod crashing because it didn't use the correct acl.conf path in the configmap.

To Reproduce
Steps to reproduce the behavior:

Change e.g. mounts + config. Observe that newly created pod still uses old config

Expected behavior
A clear and concise description of what you expected to happen.

New pod should use new config

Anything else we need to know?:

Environment details::

• Kubernetes version: v1.31.5
• emqx-operator version: 2.2.28
• Install method: helm

[Rory-Z]

Hi @lukas-mertens, could you please provide the events in the EMQX pod namespace, along with the logs for both the old EMQX pod and the EMQX Operator?

[lukas-mertens]

@Rory-Z Sadly I don't have them anymore, as we resolved the issue manually and moved on. Was it not possible to reproduce the issue in your environment using my description? If not I could try to dig deeper, but I already read some source code and came to the conclusion that it should be possible to reproduce the issue the way I explained

[Rory-Z]

@lukas-mertens I'm glad you had resolved this issue, I will try to reproduce this issue and investigate it, but I have other work now, so this work will start in next week.

[Rory-Z]

I think I know the reason, but I have no idea fix it for now. The reason is in EMQX, have so many way to change EMQX config, you can set config file, change environment variables, or use EMQX dashboard, so EMQX operator will get EMQX config by API, and set them in configMap, this is for make sure all of EMQX node has same configure.

When user change configure in spec.config.data, EMQX operator will put them to EMQX API to update EMQX config, and when it success, EMQX operator will update configMap.

When user change some field in EMQX CR which need to create new pod, EMQX operator will create new pod, but in this time, EMQX operator haven't been update EMQX config by API, that's why this issue will happen.

Even first change EMQX config and then create new pod, I think still have problem, because in old EMQX pod, the path is wrong.

@id @zmstone Do you have any ideas ?

[lukas-mertens]

I can confirm we are still seeing the issue all the time. Happens every time we make changes to our config and then redeploy.

[id]

We'll try to solve this issue in the next release.