KMS encryption for etcd

[prudnitskiy]

There are some options to encrypt the etcd config in Talos, but unfortunately, all of them are static. We are massively using KMS (Hashicorp Vault to be specific) as an encryption key provider, and I think the feature that allows users to use dynamic key provisioning may be useful to other users as well as to us. I do not see this feature in the roadmap and am happy to implement it myself as an MR, but I'd like to discuss some topics before I start to be sure I'm doing it the way Talos intended to be designed and implemented. I'd like to discuss it before I start implementing it. First of all, is there some kind of architectural decision to use static encryption only for etcd to make the encryption config stable?

Second -- I understand KMS provider setup is out of the scope of Talos setup. KMS must be set up and enabled before the cluster is deployed. It cannot be hosted inside of cluster because it creates some kind of "chicken-and-egg" problem. It provides some kind of instability and external dependency. The config is impossible to validate before it is applied. KMS permissions may be revoked later, making the key (and through this the whole cluster) unavailable. Is it an affordable trade-off for such a feature?

Thanks in advance!

[smira]

See #10899.

Every deployment takes some form of a compromise around various dimensions of security, control, performance, durability, etc.

I personally don't see any reason to use KMS (not sure what kind of a threat it mitigates), but I understand that every deployment is different, and there might be some requirements pushed which do not improve security, but are just requirements.