TLS and mTLS Fundamentals#
TLS (Transport Layer Security) encrypts traffic between two endpoints. Mutual TLS (mTLS) adds a second layer: both sides prove their identity with certificates. Understanding these is not optional for anyone building distributed systems — nearly every production failure involving “connection refused” or “certificate verify failed” traces back to a TLS misconfiguration.
How TLS Works#
A TLS handshake establishes an encrypted channel before any application data is sent. The simplified flow:
Client Server
│ │
│──── ClientHello (supported ciphers) ───→│
│ │
│←── ServerHello + Server Certificate ────│
│ │
│ Client verifies server cert against │
│ its trusted CA bundle │
│ │
│──── Key Exchange ──────────────────────→│
│ │
│←── Finished ────────────────────────────│
│ │
│ Encrypted application data flows │
└─────────────────────────────────────────┘The server presents a certificate. The client checks whether it trusts the CA that signed that certificate. If trust is established, they negotiate encryption keys. All subsequent traffic is encrypted.
The client never proves its identity in standard TLS. The server has no idea who the client is — it only knows the connection is encrypted.
Certificate Anatomy#
An X.509 certificate contains:
Certificate:
Subject: CN=api.example.com, O=Example Inc
Issuer: CN=Example CA, O=Example Inc
Validity: Not Before: 2026-01-01, Not After: 2027-01-01
Public Key: RSA 2048-bit or ECDSA P-256
SANs: DNS:api.example.com, DNS:*.example.com, IP:10.0.1.5
Key Usage: Digital Signature, Key Encipherment
Ext Key Usage: TLS Web Server Authentication
Signature: SHA256withRSA (signed by issuer's private key)Key fields:
- Subject — Who the certificate identifies. The Common Name (CN) was historically used for hostname matching but is now deprecated in favor of SANs.
- Subject Alternative Names (SANs) — The actual hostnames and IPs the certificate is valid for. This is what clients check. A certificate without SANs matching the requested hostname will be rejected by modern clients.
- Issuer — The CA that signed this certificate.
- Validity — Start and expiration dates. An expired certificate breaks everything.
- Key Usage / Extended Key Usage — What the certificate can be used for. Server certificates need
TLS Web Server Authentication. Client certificates for mTLS needTLS Web Client Authentication.
Inspect a certificate:
# From a file
openssl x509 -in cert.pem -text -noout
# From a running server
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -text -noout
# Check SANs specifically
openssl x509 -in cert.pem -noout -ext subjectAltNameChain of Trust#
Certificates form a chain:
Root CA (self-signed, in client's trust store)
└── Intermediate CA (signed by Root CA)
└── Leaf Certificate (signed by Intermediate CA, used by the server)The client starts at the leaf certificate, follows the chain up through intermediates, and checks if it reaches a root CA in its trust store. If any link in the chain is missing, expired, or untrusted, verification fails.
The server must present the full chain (leaf + intermediates). It does not send the root CA — the client already has that in its trust store. The most common TLS misconfiguration is a server presenting only the leaf certificate without the intermediate.
Create a full chain file:
cat server-cert.pem intermediate-ca.pem > fullchain.pemVerify a chain:
openssl verify -CAfile root-ca.pem -untrusted intermediate-ca.pem server-cert.pemGenerating Certificates#
Self-Signed CA for Development and Internal Services#
# Generate CA private key
openssl genrsa -out ca-key.pem 4096
# Generate CA certificate (valid 10 years)
openssl req -new -x509 -key ca-key.pem -sha256 \
-subj "/CN=Internal CA/O=MyOrg" \
-days 3650 -out ca-cert.pem
# Generate server private key
openssl genrsa -out server-key.pem 2048
# Generate server CSR with SANs
openssl req -new -key server-key.pem \
-subj "/CN=api.internal" \
-addext "subjectAltName=DNS:api.internal,DNS:*.api.internal,IP:10.0.1.5" \
-out server.csr
# Sign the server certificate with the CA
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
-CAcreateserial -out server-cert.pem -days 365 -sha256 \
-copy_extensions copyallUsing step-cli (Simpler)#
step-cli from Smallstep simplifies certificate operations:
# Create a CA
step ca init --name "Internal CA" --dns localhost --address :443
# Generate a server certificate
step certificate create api.internal server-cert.pem server-key.pem \
--ca ca-cert.pem --ca-key ca-key.pem \
--san api.internal --san "*.api.internal" --san 10.0.1.5 \
--not-after 8760h
# Generate a client certificate for mTLS
step certificate create client1 client-cert.pem client-key.pem \
--ca ca-cert.pem --ca-key ca-key.pem \
--not-after 8760hUsing cfssl#
# CA config
cat > ca-config.json << 'EOF'
{
"signing": {
"default": { "expiry": "8760h" },
"profiles": {
"server": {
"usages": ["signing", "digital signature", "key encipherment", "server auth"],
"expiry": "8760h"
},
"client": {
"usages": ["signing", "digital signature", "key encipherment", "client auth"],
"expiry": "8760h"
}
}
}
}
EOF
# Generate CA
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# Generate server cert
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=server server-csr.json | cfssljson -bare server
# Generate client cert
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=client client-csr.json | cfssljson -bare clientMutual TLS (mTLS)#
In standard TLS, only the server presents a certificate. In mTLS, both sides present certificates:
Client Server
│ │
│──── ClientHello ───────────────────────→│
│ │
│←── ServerHello + Server Certificate ────│
│←── CertificateRequest ─────────────────│ ← Server asks client for cert
│ │
│ Client verifies server cert │
│ │
│──── Client Certificate ────────────────→│ ← Client proves identity
│──── Key Exchange ──────────────────────→│
│ │
│ Server verifies client cert │
│ │
│ Both sides authenticated │
└─────────────────────────────────────────┘When to Use mTLS#
- Service-to-service communication. Services in a microservice architecture prove identity to each other. No passwords or tokens needed — the certificate is the credential.
- Zero-trust networks. Every connection must be authenticated, even within the internal network.
- API clients that are machines, not humans. IoT devices, partner integrations, agent-to-agent communication.
Configuring mTLS in Go#
// Server: require and verify client certificates
caCert, _ := os.ReadFile("ca-cert.pem")
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
serverCert, _ := tls.LoadX509KeyPair("server-cert.pem", "server-key.pem")
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: caCertPool,
MinVersion: tls.VersionTLS13,
}
server := &http.Server{
Addr: ":8443",
TLSConfig: tlsConfig,
}
server.ListenAndServeTLS("", "")// Client: present certificate to server
caCert, _ := os.ReadFile("ca-cert.pem")
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
clientCert, _ := tls.LoadX509KeyPair("client-cert.pem", "client-key.pem")
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{clientCert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS13,
}
client := &http.Client{
Transport: &http.Transport{TLSClientConfig: tlsConfig},
}
resp, _ := client.Get("https://api.internal:8443/data")Configuring mTLS in Python#
import ssl
import httpx
# Client with mTLS
ssl_context = ssl.create_default_context(cafile="ca-cert.pem")
ssl_context.load_cert_chain(certfile="client-cert.pem", keyfile="client-key.pem")
ssl_context.minimum_version = ssl.TLSVersion.TLSv1_3
client = httpx.Client(verify=ssl_context)
response = client.get("https://api.internal:8443/data")mTLS in Kubernetes with cert-manager#
cert-manager automates certificate issuance and renewal in Kubernetes:
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace \
--set crds.enabled=trueCreate a self-signed CA issuer:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca-cert
namespace: cert-manager
spec:
isCA: true
commonName: internal-ca
secretName: internal-ca-secret
issuerRef:
name: internal-ca
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca-issuer
spec:
ca:
secretName: internal-ca-secretIssue certificates for services:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: api-server-cert
namespace: my-app
spec:
secretName: api-server-tls
duration: 720h # 30 days
renewBefore: 168h # Renew 7 days before expiry
issuerRef:
name: internal-ca-issuer
kind: ClusterIssuer
dnsNames:
- api-server
- api-server.my-app.svc
- api-server.my-app.svc.cluster.local
usages:
- server auth
- client auth # Include for mTLScert-manager creates and renews the TLS Secret automatically. Mount it into your pods:
volumes:
- name: tls
secret:
secretName: api-server-tls
containers:
- name: api
volumeMounts:
- name: tls
mountPath: /tls
readOnly: trueCertificate Lifecycle#
Rotation Without Downtime#
The key to zero-downtime certificate rotation is overlapping validity periods:
- New certificate is issued while old certificate is still valid.
- Server starts serving the new certificate.
- Clients trust both old and new certificates (they trust the CA, not individual certs).
- Old certificate expires. No impact because nobody is using it.
cert-manager handles this automatically with renewBefore. The new certificate is issued before the old one expires, and Kubernetes updates the Secret. Applications watching the Secret or using file-based reloading pick up the new cert.
Monitoring Expiry#
# Check certificate expiry from a file
openssl x509 -in cert.pem -noout -enddate
# Check certificate expiry from a live server
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null | openssl x509 -noout -enddate
# Check all cert-manager certificates in a cluster
kubectl get certificates --all-namespaces -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[0].status,EXPIRY:.status.notAfter'Set up alerts for certificates expiring within 14 days. Prometheus with the cert-manager metrics or a simple CronJob that checks expiry dates works.
Troubleshooting TLS#
“certificate verify failed”#
The client does not trust the server’s certificate. Causes:
- Missing intermediate certificate. The server is sending only the leaf. Fix: concatenate the full chain.
- Self-signed CA not in trust store. The client does not have the CA certificate. Fix: add the CA to the client’s trusted roots.
- Hostname mismatch. The certificate’s SANs do not include the hostname the client connected to. Fix: regenerate the certificate with the correct SANs.
- Expired certificate. Check with
openssl x509 -in cert.pem -noout -dates.
Debug the full chain:
openssl s_client -connect api.example.com:443 -servername api.example.com -showcertsThis outputs every certificate in the chain. Check each one.
“tls: bad certificate” (mTLS)#
The server rejected the client certificate. Causes:
- Client certificate not signed by a CA the server trusts. The server’s
ClientCAspool does not include the CA that signed the client cert. - Client certificate missing
client authextended key usage. Server auth and client auth are separate usages. - Client not sending a certificate at all. The client TLS config does not have a certificate loaded.
“tls: handshake failure”#
Protocol-level incompatibility:
- TLS version mismatch. Server requires TLS 1.3, client only supports TLS 1.2. Or vice versa.
- No shared cipher suites. Server and client have no ciphers in common.
- Client sending TLS 1.0/1.1. Modern servers reject these. Upgrade the client.
# Test specific TLS versions
openssl s_client -connect api.example.com:443 -tls1_3
openssl s_client -connect api.example.com:443 -tls1_2Connection Hangs (No Response)#
Not a TLS error — the TCP connection is not reaching the server. Check:
- Firewall rules blocking the port.
- DNS resolving to the wrong IP.
- Server not listening on the expected port.
- Load balancer health check failing, causing the backend to be removed.
TLS Configuration Best Practices#
- Minimum TLS 1.2. TLS 1.0 and 1.1 are deprecated. Prefer TLS 1.3 where both sides support it.
- Use ECDSA P-256 keys. Faster than RSA, smaller certificates, equivalent security to RSA 3072.
- Short-lived certificates. 30-90 days for automated environments. Shorter validity limits the window for a compromised key.
- Automate renewal. Never rely on manual certificate renewal. Use cert-manager, ACME (Let’s Encrypt), or Vault PKI.
- Do not disable certificate verification.
InsecureSkipVerify: trueand-kflags are for debugging only. Never ship code that skips verification. - Pin to the CA, not the leaf. If you must pin certificates, pin the CA certificate or public key. Pinning leaf certificates breaks on every rotation.
Common Mistakes#
- Setting
InsecureSkipVerify: truein production. This disables all certificate verification, making the connection vulnerable to man-in-the-middle attacks. It is the TLS equivalent of removing the lock from your front door. - Not including the intermediate certificate in the chain. Works on some clients (browsers that cache intermediates) and fails on others (curl, Go, Python). Always send the full chain.
- Using CN instead of SANs for hostname matching. Chrome and Go have not checked CN for years. If the certificate has no SANs, it will be rejected.
- Forgetting to include Kubernetes service DNS names in SANs. A cert for
api-serverwill not work when accessed asapi-server.my-app.svc.cluster.local. Include all DNS forms. - Not monitoring certificate expiry. Expired certificates cause outages. By the time you notice, users are already affected. Monitor proactively.