Choosing a Secret Management Strategy#

Secrets – database credentials, API keys, TLS certificates, encryption keys – must be available to pods at runtime. At the same time, they must not be stored in plain text in git, should be rotatable without downtime, and should produce an audit trail showing who accessed what and when. No single tool satisfies every requirement, and the right choice depends on your security maturity, operational capacity, and compliance obligations.

The Core Problem#

Kubernetes Secrets are the native mechanism for delivering sensitive data to pods, but they are not a complete secret management solution. They are base64-encoded (not encrypted), stored in etcd, and visible to anyone with the appropriate RBAC permissions. The ecosystem has produced several tools to address these gaps, each with different tradeoffs around encryption, rotation, auditability, and operational complexity.

Options at a Glance#

Kubernetes Secrets (Native)#

Kubernetes Secrets store data as base64-encoded values in etcd. They are not encrypted by default, though you can enable encryption at rest by configuring an EncryptionConfiguration on the API server.

Choose native K8s Secrets when:

• You are running development or test environments where secret leakage risk is low.
• You have etcd encryption at rest enabled and strong RBAC controls.
• The application is simple and secrets rarely change.
• You want zero dependencies beyond the Kubernetes API.

Limitations to understand:

• Base64 is encoding, not encryption. Anyone who can kubectl get secret -o yaml sees the plain values.
• No rotation mechanism. Changing a secret requires manual updates and pod restarts.
• No audit trail beyond Kubernetes API audit logging (which most clusters do not enable by default).
• Secrets are stored in etcd and backed up with cluster state, so backups contain all secrets in plain text unless etcd encryption is configured.

Sealed Secrets (Bitnami)#

Sealed Secrets introduces a cluster-side controller and a client-side CLI (kubeseal). You encrypt secrets against the cluster’s public key, producing a SealedSecret CRD that is safe to commit to git. The controller in the cluster decrypts them into regular Kubernetes Secrets.

Choose Sealed Secrets when:

• You follow a GitOps workflow and need secrets stored in the same repository as your manifests.
• Your team is small to medium and manages one or a few clusters.
• You want a simple, low-overhead solution without external dependencies.
• Secret rotation is infrequent and handled manually.

Limitations to understand:

• Single cluster scope. Each cluster has its own key pair, so sealed secrets are not portable between clusters without re-encryption.
• No dynamic secrets. You seal a static value; there is no mechanism to generate time-limited credentials.
• No rotation. When a credential changes, you must re-seal and redeploy.
• Key rotation for the sealing key itself requires planning – the controller keeps old keys to decrypt existing SealedSecrets, but the process needs care.

External Secrets Operator (ESO)#

The External Secrets Operator syncs secrets from external stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault, and others) into Kubernetes Secrets. You define an ExternalSecret CRD that references a key in the external store, and ESO creates and updates the corresponding K8s Secret.

Choose External Secrets Operator when:

• You already use a cloud-managed secret store and want secrets injected into Kubernetes without changing application code.
• Centralized secret management across multiple clusters is a requirement.
• You want secrets managed by a platform team in an external store while application teams consume them as K8s Secrets.
• Automatic sync (rotation) from the external store is important.

Limitations to understand:

• Adds a dependency. If ESO or the external store is unavailable, new pods cannot get secrets (existing secrets persist).
• Sync delay. There is a configurable interval between the external store update and the K8s Secret update (default is often 1 hour).
• Secrets still exist as K8s Secrets in etcd after sync, so etcd encryption at rest is still important.
• Debugging requires understanding both the ESO layer and the external store.

HashiCorp Vault#

Vault is a full-featured secret management platform. It provides static secrets, dynamic secrets (generating short-lived database credentials, cloud IAM credentials, PKI certificates on demand), encryption as a service, and comprehensive audit logging.

Choose Vault when:

• You have enterprise security requirements: dynamic database credentials, automated PKI, encryption as a service.
• Audit trail requirements demand knowing exactly who accessed which secret and when.
• You manage multiple clusters and need a centralized secret authority.
• You need dynamic secrets – credentials that exist only for the lifetime of a pod and are automatically revoked.
• Regulatory compliance requires key management with HSM backing.

Limitations to understand:

• Significant operational complexity. Vault is another critical system to deploy, monitor, back up, unseal (or auto-unseal), and upgrade.
• Enterprise features (namespaces, Sentinel policies, HSM support, replication) require a paid license.
• Learning curve for operators and developers. The Vault API, policies, and authentication methods take time to learn.
• Vault downtime can be catastrophic if applications cannot start without fetching secrets.

SOPS (Mozilla)#

SOPS encrypts values in-place within YAML, JSON, or ENV files using encryption keys from AWS KMS, GCP KMS, Azure Key Vault, age, or PGP. The file structure remains readable (keys are visible, values are encrypted), making diffs meaningful.

Choose SOPS when:

• You use Flux for GitOps (Flux has native SOPS decryption support built in).
• You want encrypted secret files in git without introducing CRDs or controllers.
• Your team is comfortable managing encryption keys (KMS or age).
• You prefer a file-based workflow over CRD-based approaches.

Limitations to understand:

• Manual process. Encrypting and decrypting files requires running sops commands or having CI/CD tooling to handle it.
• No dynamic secrets. You encrypt static values.
• No rotation mechanism beyond manually updating and re-encrypting files.
• Key management responsibility falls on you. If using age keys, losing the private key means losing access to all encrypted files.

Cloud KMS Direct#

Applications read secrets directly from cloud provider secret stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) using the provider’s SDK, without any Kubernetes intermediary.

Choose cloud KMS direct when:

• Your application is cloud-native and already uses the cloud provider’s SDK.
• You are willing to couple the application to a specific cloud provider.
• You want the simplest possible architecture with the fewest moving parts.
• The application needs to use provider-specific features (secret versioning, cross-region replication).

Limitations to understand:

• Vendor lock-in. Your application code must know how to authenticate and fetch from the specific provider.
• No portability. Moving to another cloud or on-prem requires rewriting secret fetching logic.
• Authentication requires workload identity or IAM roles for service accounts, adding cloud-specific configuration.
• Local development and testing becomes harder because you need cloud credentials or must mock the provider.

Comparison by Decision Criteria#

If your primary concern is simplicity: Native K8s Secrets (with etcd encryption enabled) or Sealed Secrets. Both have minimal moving parts.

If your primary concern is GitOps compatibility: Sealed Secrets or SOPS. Both produce encrypted artifacts safe to commit. External Secrets Operator works with GitOps too (you commit the ExternalSecret CRD, not the secret value).

If your primary concern is security and compliance: Vault. Nothing else provides dynamic secrets, comprehensive audit logging, and encryption as a service in a single tool.

If your primary concern is multi-cluster consistency: External Secrets Operator with a centralized backend (Vault, AWS Secrets Manager). One source of truth, synced to all clusters.

If your primary concern is cost: Native K8s Secrets and Sealed Secrets cost nothing. SOPS is free. Vault OSS is free but costs operational time. Cloud secret stores charge per API call ($0.05 per 10,000 API calls for AWS Secrets Manager, plus $0.40/secret/month).

Combination Patterns#

Most production environments use more than one tool.

External Secrets Operator + Vault: Vault serves as the secret backend. ESO syncs Vault secrets into Kubernetes Secrets. Applications consume standard K8s Secrets without knowing Vault exists. This gives you Vault’s power without requiring every application to integrate the Vault SDK.

Sealed Secrets for config + Vault for credentials: Use Sealed Secrets for non-sensitive configuration that still should not be in plain text (internal URLs, feature flags). Use Vault for actual credentials (database passwords, API keys, TLS certificates).

SOPS + Flux for GitOps + ESO for dynamic secrets: Flux decrypts SOPS-encrypted files during reconciliation for static configuration. ESO handles secrets that need rotation or come from a centralized store.

Migration Path: Progressive Secret Maturity#

Organizations typically progress through these stages:

Stage 1 – Native K8s Secrets: Get the application running. Enable etcd encryption at rest. Restrict RBAC access to secrets. This is acceptable for early-stage projects and development clusters.

Stage 2 – Sealed Secrets or SOPS: Adopt GitOps. Stop passing secrets through CI/CD pipelines in plain text. Encrypted secrets in git means your deployment manifests are self-contained and auditable through git history.

Stage 3 – External Secrets Operator + Cloud Secret Store: Centralize secrets in a managed store. Multiple clusters sync from one source. Platform teams manage secrets; application teams consume them.

Stage 4 – Vault: Introduce dynamic secrets for databases (credentials generated per pod, automatically revoked). Deploy PKI management for internal mTLS. Enable full audit logging. This stage is appropriate when compliance or security requirements demand it, or when the operational team has the capacity to manage Vault.

Not every organization needs Stage 4. Many operate effectively at Stage 2 or 3 for years. The key is matching the tool to your actual security requirements and operational capacity, not aspirational ones.