The Secret Zero Problem#

Every secrets management system has the same fundamental challenge: you need a secret to access your secrets. Your Vault token is itself a secret. Your AWS credentials for SSM Parameter Store are themselves secrets. This is the “secret zero” problem – there is always one secret that must be bootstrapped outside the system.

Understanding this helps you make pragmatic choices. No tool eliminates all risk. The goal is to reduce the blast radius and make rotation possible.

The Progression#

Secrets management is not a binary choice. It is a progression tied to your project maturity.

Level 1: kubectl create secret (POC)#

Good for: local development, proof of concept, single developer.

kubectl create secret generic app-secrets \
  --from-literal=database-url="postgres://user:pass@db:5432/mydb" \
  --from-literal=api-key="sk-abc123" \
  -n default

Reference in a pod:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: database-url

Why this breaks down: Secrets are not in Git, so there is no history, no review process, and no way to recreate the cluster without someone manually running kubectl create secret again. If that person leaves the team, you have a bus factor of one.

Level 2: Sealed Secrets (Small Teams)#

Good for: teams of 2-10, single cluster, secrets that change infrequently.

Sealed Secrets lets you encrypt secrets and store them in Git. Only the Sealed Secrets controller running in the cluster can decrypt them.

Install:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system \
  --wait

Install the CLI:

# macOS
brew install kubeseal

Create a sealed secret:

# Create a regular secret manifest (do NOT apply it)
kubectl create secret generic app-secrets \
  --from-literal=database-url="postgres://user:pass@db:5432/mydb" \
  --dry-run=client -o yaml > secret.yaml

# Encrypt it
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

# Now sealed-secret.yaml is safe to commit to Git
rm secret.yaml
git add sealed-secret.yaml && git commit -m "Add encrypted app secrets"

Apply the sealed secret to the cluster:

kubectl apply -f sealed-secret.yaml
# The controller decrypts it and creates the actual Secret

Secret zero for Sealed Secrets: The encryption key pair is generated automatically in the cluster. Back it up. If you lose it (cluster rebuild without backup), you cannot decrypt existing sealed secrets.

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-backup.yaml

Level 3: External Secrets Operator (Multi-Cloud / Growing Teams)#

Good for: teams using cloud provider secret stores, multi-cluster setups, compliance requirements.

The External Secrets Operator (ESO) syncs secrets from external providers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault) into Kubernetes Secrets.

Install:

helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --wait

Create a SecretStore that connects to AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-access-key

Create an ExternalSecret that pulls a specific secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: app-secrets
  data:
    - secretKey: database-url
      remoteRef:
        key: myapp/database-url

ESO creates and updates the Kubernetes Secret automatically. When you rotate the secret in AWS, ESO picks up the change within refreshInterval.

Secret zero for ESO: The AWS credentials in the aws-credentials secret. On EKS, use IRSA (IAM Roles for Service Accounts) to eliminate this entirely.

Level 4: HashiCorp Vault with CSI Driver (Enterprise)#

Good for: enterprises with compliance requirements, dynamic secrets, fine-grained access policies.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm upgrade --install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.dev.enabled=true \
  --wait

The CSI driver mounts secrets directly into pods as files:

volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault-db-creds

Vault’s real power is dynamic secrets – it generates short-lived database credentials on demand, so there are no long-lived passwords to rotate or leak. This requires significantly more setup and operational knowledge.

Decision Matrix#

The Recommendation#

Start with Sealed Secrets. It solves the biggest pain point (secrets not in Git) with minimal operational overhead. You can set it up in 15 minutes and it requires almost no ongoing maintenance.

Graduate to External Secrets Operator when any of these happen: you go multi-cloud, you need automatic secret rotation, compliance requires an audited secret store, or your team outgrows manually creating sealed secrets.

Add Vault when you need dynamic secrets (short-lived database credentials), fine-grained access policies (this team can read these secrets but not those), or you are in a regulated industry that requires a dedicated secrets management platform.

Do not start with Vault for a POC. The operational overhead will slow you down and distract from building the actual product.