Why Rotation Matters#

A credential that never changes is a credential waiting to be exploited. Leaked credentials appear in git history, log files, CI build outputs, developer laptops, and third-party SaaS tools. If a database password has been the same for two years, every person who has ever had access to it still has access – former employees, former contractors, compromised CI systems.

Regular rotation limits the blast radius. A credential that rotates every 24 hours is only useful for 24 hours after compromise. Compliance frameworks (PCI-DSS, SOC2, HIPAA) mandate rotation schedules. But compliance aside, rotation is a pragmatic defense: assume credentials will leak and make the leak time-limited.

Rotation Strategies#

Manual Rotation#

A human generates a new password, updates it in the database, updates all application configurations, and restarts services. This is the most common approach and the most error-prone. It causes downtime when coordination fails – update the database password but forget one of the five services that use it, and that service goes down at 3 AM.

Manual rotation is acceptable for break-glass credentials (the root password you use once a year in an emergency). It is not acceptable for credentials used by running services.

Scheduled Rotation#

Automated rotation on a fixed schedule: every 30, 60, or 90 days, a script generates a new credential and updates all consumers. This eliminates the human error of manual rotation but still has a window of disruption. The moment between “new credential is active” and “all consumers are updated” is a risk window.

Scheduled rotation works well when combined with the dual-credential pattern (described below) to eliminate the disruption window.

Dynamic Secrets#

The credential is generated on demand and exists only for the duration of its lease. A service requests a database credential from Vault, receives a unique username and password valid for 1 hour, uses it, and Vault automatically revokes it at expiry. No credential is ever stored permanently. No credential is shared between services. If one is compromised, revoke it without affecting any other service.

Dynamic secrets are the gold standard but require infrastructure investment (Vault, credential management pipelines). Use them for database credentials, cloud provider access keys, and any credential where the target system supports programmatic user creation.

The Dual-Credential Pattern#

The dual-credential pattern eliminates downtime during rotation. It works for any credential type where the target system can accept two valid credentials simultaneously.

Step 1: Generate new credential. Create the new password, API key, or certificate. At this point, both the old and new credentials are valid. The target system (database, API, service) accepts both.

Step 2: Update consumers. Roll out the new credential to all consumers via rolling deployment, config update, or secret sync. This takes time – minutes to hours depending on the number of consumers and deployment strategy.

Step 3: Verify. Confirm that all consumers are using the new credential. Check application logs, connection pool metrics, or directly query the database for active sessions using the new role.

Step 4: Revoke old credential. Once all consumers are verified on the new credential, remove the old one. The target system now accepts only the new credential.

The key requirement is that both credentials must be valid simultaneously during the transition. For databases, this means two valid roles with the same permissions. For API keys, this means two active keys on the same account. For TLS certificates, both the old and new certificate are valid (they have overlapping validity periods by definition since the old one has not expired yet when the new one is issued).

Database Credential Rotation#

PostgreSQL Manual Dual-Credential Rotation#

-- Step 1: Create new role with same permissions
CREATE ROLE app_user_v2 WITH LOGIN PASSWORD 'new-secure-password';
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_user_v2;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user_v2;

-- Step 2: Update application connection strings (rolling update)
-- ... deploy with new credentials ...

-- Step 3: Verify no connections using old role
SELECT usename, count(*) FROM pg_stat_activity
WHERE usename IN ('app_user_v1', 'app_user_v2')
GROUP BY usename;

-- Step 4: Revoke old role
REVOKE ALL ON ALL TABLES IN SCHEMA public FROM app_user_v1;
DROP ROLE app_user_v1;

Vault Dynamic Database Secrets#

Vault eliminates manual rotation entirely. It creates a unique credential per lease and revokes it automatically:

# Configure the database connection
vault write database/config/production-db \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production" \
  allowed_roles="app-role" \
  username="vault_admin" \
  password="vault_admin_password"

# Define the role (what Vault creates for each lease)
vault write database/roles/app-role \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Each application instance requests its own credential:

vault read database/creds/app-role
# Returns: username=v-approle-app-role-abc123, password=A1B2C3..., lease_id=..., lease_duration=3600

The credential is unique to this instance and valid for 1 hour. The application renews the lease before expiry or requests a new credential. When the lease expires or is revoked, Vault executes the revocation_statements and drops the role.

AWS RDS with Secrets Manager#

AWS Secrets Manager supports automatic rotation with a Lambda function:

aws secretsmanager rotate-secret \
  --secret-id production/db/credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-db-secret \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

The rotation Lambda implements the dual-credential pattern: it creates a new credential, tests it, updates the secret, and cleans up the old credential. AWS provides template Lambdas for RDS, Redshift, and DocumentDB.

API Key Rotation#

API key rotation follows the dual-credential pattern with a grace period:

  1. Issue new key through the API provider’s console or API
  2. Update consumers – deploy the new key to all services that use it
  3. Grace period – keep the old key active for 24 to 72 hours to account for deployment propagation, cached configurations, and services that restart on their own schedule
  4. Revoke old key after the grace period and verify no services are still using it

The grace period duration depends on how quickly you can update all consumers. If all consumers are in a single Kubernetes cluster with rolling deployments, 1 hour might suffice. If consumers span multiple teams, environments, and third-party integrations, 72 hours is safer.

Track API key usage by key identifier. Most API providers show last-used timestamps per key. Wait until the old key shows zero usage before revoking it.

TLS Certificate Rotation#

TLS certificates rotate naturally through their lifecycle. The key is automating issuance so renewal happens well before expiry.

cert-manager handles this automatically. When a Certificate resource is created, cert-manager issues the certificate, stores it as a Kubernetes Secret, and renews it when two-thirds of its lifetime has elapsed (or at the renewBefore threshold).

Vault PKI with short-lived certificates: issue certificates valid for 24 hours. The application requests a new certificate from Vault before the current one expires. No revocation infrastructure is needed because the certificate expires before a revocation workflow could complete.

SPIFFE/SPIRE rotates workload certificates automatically, typically every hour. The SPIRE agent delivers a new SVID to the workload before the current one expires. The workload does not need to know about certificate management at all.

Kubernetes Secret Rotation#

External Secrets Operator#

The External Secrets Operator syncs secrets from external stores (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets. The refreshInterval controls how often it checks for updates:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db-credentials
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-backend
  target:
    name: app-db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: database/creds/app-role
        property: password

When Vault rotates the credential, the External Secrets Operator picks up the change within 5 minutes and updates the Kubernetes Secret.

Volume-Mounted vs Environment Variable Secrets#

Volume-mounted secrets are updated by the kubelet automatically when the underlying Secret changes. The update is eventually consistent – the kubelet sync period is configurable but defaults to roughly 1 minute. The application must watch the file or re-read it periodically to pick up changes.

Environment variable secrets are injected at pod startup and never updated. If the Secret changes, the pod must be restarted to see the new value. This makes environment variable secrets incompatible with automatic rotation. Use volume mounts instead.

Reloader for Automatic Restarts#

Stakater Reloader watches for Secret and ConfigMap changes and triggers rolling restarts of Deployments that reference them:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - name: myapp
          envFrom:
            - secretRef:
                name: app-db-credentials

When the app-db-credentials Secret changes (updated by External Secrets Operator after Vault rotation), Reloader detects the change and triggers a rolling restart of the myapp Deployment. The new pods start with the updated credentials.

This is the pragmatic solution for applications that read credentials at startup and do not support runtime credential refresh. The combination of External Secrets Operator (sync from Vault) plus Reloader (restart on change) provides fully automated rotation with zero application code changes.

Monitoring Rotation Health#

Track these metrics to ensure rotation is working:

Secret age: alert when any secret is older than the rotation policy allows. If the policy is 30-day rotation, alert at 25 days. Use labels or annotations on Kubernetes Secrets with the last rotation timestamp.

Last rotation timestamp: record when each secret was last rotated. Compare against the expected schedule. A secret that should rotate daily but last rotated 3 days ago indicates a broken rotation pipeline.

Failed rotation attempts: rotation can fail for many reasons – database unreachable, permission denied creating a new role, Vault seal status. Every rotation attempt should log success or failure. Alert on any failure immediately because the clock is ticking on the current credential’s lifetime.

Active credential count: in the dual-credential pattern, monitor how many valid credentials exist at any time. After rotation completes, there should be exactly one. If there are two for longer than the expected grace period, the old credential was not cleaned up.

Common Gotchas#

Rotating a database password while connections are open. Existing database connections authenticated with the old password continue to work. The connection was already established and authenticated – changing the password does not invalidate open connections. New connections require the new password. This means there is a natural grace period: existing connections use old credentials until they are recycled by the connection pool. Configure your connection pool to recycle connections periodically (every 30 minutes) so that stale credentials are not held indefinitely.

Rotating a shared secret used by multiple services. If 10 services use the same database password and you rotate it, all 10 must update before the old password is revoked. A race condition exists: if any service misses the update, it breaks. The dual-credential pattern solves this, but the real fix is to stop sharing credentials. Each service should have its own credential (Vault dynamic secrets achieve this automatically). When every service has a unique credential, rotating one does not affect the others.

Backward OAuth2 and OIDC for Infrastructure Zero Trust Networking Forward