securely save api keys in encrypted secrets
The lastest version 1.29 of kubernetes better supports encrypted secrets:
KMS v2 encryption at rest generally available (SIG Auth) One of the first things to consider when securing a Kubernetes cluster is encrypting persisted API data at rest. KMS provides an interface for a provider to utilize a key stored in an external key service to perform this encryption. With the Kubernetes v1.29, KMS v2 has become a stable feature bringing numerous improvements in performance, key rotation, health check & status, and observability. These enhancements provide users with a reliable solution to encrypt all resources in their Kubernetes clusters. You can read more about this in KEP-3299.
It is recommended to use KMS v2. KMS v1 feature gate is disabled by default. You will have to opt in to continue to use it.
IMPORTANT: to stay compatible with other execution environments like SLURM/HPC/Docker Compose, the secrets should be given to the containers as environment variables, not mounted in the container file system. So it should be handled like the SHARED_FOLDER_PATH