Securing etcd#

etcd is the single most critical component in a Kubernetes cluster. It stores everything: pod specs, secrets, configmaps, RBAC rules, service account tokens, and all cluster state. By default, Kubernetes secrets are stored in etcd as base64-encoded plaintext. Anyone with read access to etcd has read access to every secret in the cluster. Securing etcd is not optional.

Why etcd Is the Crown Jewel#

Run this against an unencrypted etcd and you will see why:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

The output contains the secret value in plaintext (base64-encoded, which is trivially decoded). Database passwords, API keys, TLS private keys – all readable by anyone who can access etcd directly.

Encryption at Rest#

Kubernetes supports encrypting secrets before they are written to etcd through an EncryptionConfiguration file.

Create the Encryption Configuration#

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Generate a 32-byte encryption key:

head -c 32 /dev/urandom | base64

The secretbox provider uses XSalsa20-Poly1305 and is recommended over aescbc because it provides authenticated encryption. The identity provider at the end is a fallback that allows reading secrets that were stored before encryption was enabled.

Apply the Configuration#

Save the file to /etc/kubernetes/enc/encryption-config.yaml on every control plane node, then add it to the API server manifest:

# In /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
      volumeMounts:
        - name: enc
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: enc
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

After the API server restarts, new secrets are encrypted. Existing secrets remain unencrypted until they are rewritten:

# Re-encrypt all existing secrets
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

Verify encryption is working:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | head -20

You should see k8s:enc:secretbox:v1:key1 at the beginning of the data, followed by encrypted bytes instead of readable text.

Key Rotation#

To rotate encryption keys, add a new key as the first entry in the providers list, keep the old key as the second entry, update the API server, then re-encrypt all secrets:

providers:
  - secretbox:
      keys:
        - name: key2
          secret: <new-base64-encoded-32-byte-key>
        - name: key1
          secret: <old-base64-encoded-32-byte-key>
  - identity: {}

After re-encrypting all secrets with kubectl replace, remove the old key and restart the API server again. Rotate keys at least annually.

etcd TLS#

etcd must use TLS for both client-to-server and peer-to-peer communication. Managed Kubernetes services handle this automatically. For self-managed clusters with kubeadm, TLS is configured during cluster initialization, but verify it:

# Check etcd is using TLS
ps aux | grep etcd | grep -o '\-\-[a-z-]*tls[a-z-]*=[^ ]*'

You should see these flags:

--cert-file=/etc/kubernetes/pki/etcd/server.crt
--key-file=/etc/kubernetes/pki/etcd/server.key
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--client-cert-auth=true
--peer-client-cert-auth=true

The --client-cert-auth=true flag is critical – it means etcd requires clients to present a valid certificate signed by the CA. Without it, anyone who can reach the etcd port can read all cluster data.

Access Control#

Network-Level Isolation#

etcd should only be accessible from the API server. On the control plane nodes, use firewall rules or network policies:

# iptables: only allow the API server to reach etcd
iptables -A INPUT -p tcp --dport 2379 -s <api-server-ip> -j ACCEPT
iptables -A INPUT -p tcp --dport 2379 -j DROP
iptables -A INPUT -p tcp --dport 2380 -s <peer-etcd-ips> -j ACCEPT
iptables -A INPUT -p tcp --dport 2380 -j DROP

Port 2379 is the client port (API server connections). Port 2380 is the peer port (etcd cluster replication). Both should be locked down.

etcd Authentication#

etcd supports its own user and role-based authentication on top of TLS:

# Enable authentication
etcdctl user add root --new-user-password="<password>"
etcdctl auth enable

# Create a read-only role for monitoring
etcdctl role add monitoring
etcdctl role grant-permission monitoring read --prefix /
etcdctl user add monitor --new-user-password="<password>"
etcdctl user grant-role monitor monitoring

In practice, most Kubernetes deployments rely on TLS client certificates for authentication rather than etcd’s built-in auth. Both layers can be used together for defense in depth.

Backup Security#

etcd backups contain all cluster secrets. An unprotected backup is equivalent to full cluster compromise.

# Create an encrypted backup
ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Encrypt the backup file before storing it
gpg --symmetric --cipher-algo AES256 /tmp/etcd-snapshot.db
rm /tmp/etcd-snapshot.db

Store encrypted backups in a location with strict access controls. Use server-side encryption on S3 buckets or equivalent. Restrict IAM policies so only the backup service account and disaster recovery team can access the backups.

Monitoring for Unauthorized Access#

Watch etcd metrics for signs of unauthorized access:

# Check for auth failures
etcdctl endpoint status --write-out=table

# Monitor etcd logs for authentication errors
journalctl -u etcd | grep -i "auth\|denied\|rejected\|failed"

Alert on unusual patterns: high read rates on /registry/secrets, connections from unexpected IPs, and authentication failures. These are early indicators of compromise or misconfiguration.