HashiCorp Vault on Kubernetes#

Vault centralizes secret management with dynamic credentials, encryption as a service, and fine-grained access control. On Kubernetes, workloads authenticate using service accounts and pull secrets without hardcoding anything.

Installation with Helm#

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Dev Mode (Single Pod, In-Memory)#

Automatically initialized and unsealed, stores everything in memory, loses all data on restart. Root token is root. Never use this in production.

helm upgrade --install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set server.dev.enabled=true \
  --set injector.enabled=true

Production Mode (HA with Integrated Raft Storage)#

Run Vault in HA mode with Raft consensus – a 3-node StatefulSet with persistent storage.

# values-prod.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }
        service_registration "kubernetes" {}
  dataStorage:
    enabled: true
    size: 10Gi
injector:
  enabled: true

After install, initialize and unseal manually:

kubectl exec -n vault vault-0 -- vault operator init -key-shares=5 -key-threshold=3
# Save the unseal keys and root token securely

kubectl exec -n vault vault-0 -- vault operator unseal <key-1>
kubectl exec -n vault vault-0 -- vault operator unseal <key-2>
kubectl exec -n vault vault-0 -- vault operator unseal <key-3>

Join other nodes to the Raft cluster:

kubectl exec -n vault vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec -n vault vault-2 -- vault operator raft join http://vault-0.vault-internal:8200

Kubernetes Auth Method#

Pods authenticate to Vault using their Kubernetes service account tokens. Vault verifies the token with the Kubernetes API.

kubectl exec -n vault vault-0 -- vault auth enable kubernetes

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Create a role binding a service account to a Vault policy:

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=payments \
  policies=app-policy \
  ttl=1h

Any pod running as app-sa in namespace payments can now authenticate and receive tokens scoped to app-policy.

Secret Engines#

KV v2 (Static Secrets)#

kubectl exec -n vault vault-0 -- vault secrets enable -path=secret kv-v2
kubectl exec -n vault vault-0 -- vault kv put secret/payments/db \
  username="payments-user" password="s3cur3-p4ss"

Database Dynamic Credentials#

Vault generates short-lived database credentials on demand and revokes them when the lease expires.

kubectl exec -n vault vault-0 -- vault secrets enable database

kubectl exec -n vault vault-0 -- vault write database/config/payments-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="payments-readonly" \
  connection_url="postgresql://{{username}}:{{password}}@payments-postgres:5432/payments?sslmode=disable" \
  username="vault-admin" password="admin-password"

kubectl exec -n vault vault-0 -- vault write database/roles/payments-readonly \
  db_name=payments-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" max_ttl="24h"

Vault Policies#

Policies follow a deny-by-default model. Note the secret/data/ prefix for KV v2 – the API path includes /data/ even though the CLI uses vault kv put secret/payments/db.

kubectl exec -n vault vault-0 -- vault policy write app-policy - <<EOF
path "secret/data/payments/*" {
  capabilities = ["read"]
}
path "database/creds/payments-readonly" {
  capabilities = ["read"]
}
EOF

Vault Agent Injector (Sidecar Injection)#

The injector runs as an admission webhook. Annotate pods and it injects an init container (pre-populates secrets) and a sidecar (keeps them refreshed).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "app-role"
        vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/payments/db"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "secret/data/payments/db" -}}
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASS="{{ .Data.data.password }}"
          {{- end }}
    spec:
      serviceAccountName: app-sa
      containers:
      - name: app
        image: payments-api:latest
        command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && ./start.sh"]

Secrets land at /vault/secrets/<name>. The template uses Go template syntax to format output as env files, JSON, or connection strings.

CSI Secret Store Driver#

An alternative to the sidecar. The Secrets Store CSI driver mounts secrets as a volume. Create a SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: payments
spec:
  provider: vault
  parameters:
    roleName: "app-role"
    vaultAddress: "http://vault.vault.svc:8200"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/payments/db"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/payments/db"
        secretKey: "password"

Use the Agent Injector when you need template rendering or dynamic credential auto-renewal. Use the CSI driver when you want to sync Vault secrets to Kubernetes Secrets for env vars, or want to avoid sidecar overhead.

Common Pitfalls#

  1. KV v2 path confusion. The API path is secret/data/payments/db, the CLI path is secret/payments/db. Policies must use the API path.
  2. Unsealed state lost on restart. Raft persists data, but Vault must be unsealed after every pod restart. Use auto-unseal with a cloud KMS in production.
  3. Service account token expiry. Kubernetes 1.24+ uses bound tokens with expiry. Vault 1.9+ handles this correctly.
  4. Injector needs the service account. The pod must specify serviceAccountName matching the Vault role binding. The default service account will not work unless explicitly bound.