The TLS Handshake#

Every HTTPS connection starts with a TLS handshake that establishes encryption parameters and verifies the server’s identity. The simplified flow for TLS 1.2:

Client                              Server
  |── ClientHello ──────────────────>|   (supported versions, cipher suites, random)
  |<────────────────── ServerHello ──|   (chosen version, cipher suite, random)
  |<──────────────── Certificate  ──|   (server's certificate chain)
  |<───────────── ServerKeyExchange ─|   (key exchange parameters)
  |<───────────── ServerHelloDone  ──|
  |── ClientKeyExchange ───────────>|   (client's key exchange contribution)
  |── ChangeCipherSpec ────────────>|   (switching to encrypted communication)
  |── Finished ────────────────────>|   (encrypted verification)
  |<──────────── ChangeCipherSpec ──|
  |<──────────────────── Finished ──|
  |<═══════ Encrypted traffic ═════>|

TLS 1.3 simplifies this significantly. The client sends its key share in the ClientHello, allowing the handshake to complete in a single round trip. TLS 1.3 also removed insecure cipher suites and compression, eliminating entire classes of vulnerabilities (BEAST, CRIME, POODLE).

Certificate Chains#

A TLS certificate is not trusted on its own. Trust flows from root CAs that are pre-installed in operating systems and browsers, through intermediate CAs, down to the leaf certificate on your server.

Root CA (DigiCert Global Root G2)
  └── Intermediate CA (DigiCert SHA2 Extended Validation Server CA)
       └── Leaf Certificate (example.com)

The server must send the leaf certificate and all intermediate certificates. Root certificates are not sent – the client already has them. If the intermediate is missing, some clients can fetch it (browsers often do), but others cannot (curl, programmatic HTTP clients, mobile apps). This creates the infuriating situation where a site works in Chrome but fails in curl.

To verify a certificate chain:

# Download and display the full chain from a server
openssl s_client -connect example.com:443 -showcerts </dev/null 2>/dev/null

# Verify the chain
openssl verify -CAfile root-ca.pem -untrusted intermediate.pem leaf.pem

When configuring a web server, concatenate the certificates in order – leaf first, intermediates after:

cat leaf.pem intermediate.pem > fullchain.pem

Certificate Types#

DV (Domain Validation): proves you control the domain. Issued in seconds through automated challenges. Let’s Encrypt provides these for free. Sufficient for most use cases.

OV (Organization Validation): proves the organization’s identity. Requires business documentation. Takes days. The certificate contains the organization name, but browsers do not display it differently from DV – making OV mostly a marketing distinction.

EV (Extended Validation): the most rigorous validation. Used to trigger the green bar with the company name in browsers. Most browsers removed this visual distinction in 2019-2020, making EV certificates functionally equivalent to DV for end users. Their only remaining use is edge cases where a partner or regulator specifically requires EV.

For nearly all production use cases, DV certificates from Let’s Encrypt are the correct choice. They are free, automated, and trusted by every browser and client.

Let’s Encrypt and ACME#

Let’s Encrypt uses the ACME protocol for automated certificate issuance. Two challenge types prove domain control:

HTTP-01: Let’s Encrypt gives you a token. You serve it at http://yourdomain.com/.well-known/acme-challenge/TOKEN. Let’s Encrypt fetches it and issues the certificate. Requires port 80 open and an HTTP server running. Cannot issue wildcard certificates.

DNS-01: Let’s Encrypt gives you a token. You create a TXT record _acme-challenge.yourdomain.com with the token value. Let’s Encrypt queries DNS and issues the certificate. Does not require port 80. Can issue wildcard certificates. Requires API access to your DNS provider for automation.

# HTTP-01 with certbot standalone
certbot certonly --standalone -d example.com -d www.example.com

# DNS-01 with certbot and Route53 plugin
certbot certonly --dns-route53 -d example.com -d '*.example.com'

Let’s Encrypt certificates expire after 90 days by design – short lifetimes limit the damage from compromised keys and force automation. Set up a cron job or systemd timer for renewal:

# Test renewal (does not actually renew)
certbot renew --dry-run

# Add to crontab (runs twice daily, only renews if within 30 days of expiry)
0 0,12 * * * certbot renew --quiet --post-hook "systemctl reload nginx"

cert-manager in Kubernetes#

cert-manager automates TLS certificate management inside Kubernetes. It watches for Certificate resources and Ingress annotations, issues certificates from configured issuers, and handles renewal automatically.

Set up a ClusterIssuer for Let’s Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

Request a certificate through an Ingress annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - app.example.com
      secretName: app-tls-cert
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

cert-manager creates a Certificate resource, performs the ACME challenge, stores the certificate and key in the app-tls-cert Secret, and renews it before expiry.

Check certificate status:

kubectl get certificates -A
kubectl describe certificate app-tls-cert
kubectl get challenges -A    # shows pending ACME challenges
kubectl get orders -A        # shows certificate orders

Self-Signed Certificates#

For internal services, development, and testing, self-signed certificates avoid the need for a public CA. The tradeoff is that clients do not trust them by default.

# Generate a self-signed certificate with SAN
openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt \
  -days 365 -nodes \
  -subj "/CN=internal-api.local" \
  -addext "subjectAltName=DNS:internal-api.local,DNS:*.internal.local,IP:10.0.0.5"

Modern TLS clients require Subject Alternative Names (SANs). The Common Name (CN) field alone is no longer sufficient and is deprecated. Always include SAN entries for every hostname and IP the certificate should cover.

To add a self-signed CA to trust stores:

# macOS
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ca.crt

# Ubuntu/Debian
sudo cp ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# RHEL/CentOS
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

mTLS (Mutual TLS)#

Standard TLS authenticates only the server. Mutual TLS (mTLS) requires both sides to present certificates, providing client authentication without passwords or API keys.

mTLS is used in service meshes (Istio, Linkerd) to authenticate service-to-service traffic, in zero-trust architectures where every connection must be authenticated, and for machine-to-machine API access.

In a service mesh, the sidecar proxy handles mTLS transparently – your application code sends plain HTTP, and the proxy encrypts it with mTLS to the destination’s proxy. When implementing mTLS manually, both the client and server need certificates signed by a trusted CA:

# Server verifies client certificate
openssl s_server -cert server.crt -key server.key \
  -CAfile ca.crt -Verify 1 -port 8443

# Client connects with its certificate
curl --cert client.crt --key client.key --cacert ca.crt \
  https://internal-api.local:8443/health

Cipher Suites and TLS Versions#

TLS 1.3 (released 2018) is the current standard. It removed all insecure cipher suites, reduced the handshake to one round trip, and eliminated unnecessary complexity. TLS 1.2 is still widely supported and acceptable. TLS 1.0 and 1.1 are deprecated and should be disabled.

A recommended NGINX configuration:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;  # Let the client choose in TLS 1.3
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
ssl_stapling on;
ssl_stapling_verify on;

Key principles: use ECDHE for key exchange (provides forward secrecy), AES-GCM for encryption (hardware-accelerated on modern CPUs), and disable anything with CBC, RC4, DES, or MD5.

Debugging TLS#

Check what a server presents:

# Connect and show certificate details
openssl s_client -connect example.com:443 </dev/null 2>/dev/null | \
  openssl x509 -noout -text

# Check expiry date
openssl s_client -connect example.com:443 </dev/null 2>/dev/null | \
  openssl x509 -noout -dates

# Show the full certificate chain
openssl s_client -connect example.com:443 -showcerts </dev/null

# Test specific TLS version
openssl s_client -connect example.com:443 -tls1_2
openssl s_client -connect example.com:443 -tls1_3

# Check which cipher was negotiated
openssl s_client -connect example.com:443 </dev/null 2>/dev/null | \
  grep "Cipher is"

Common TLS errors and their causes:

Certificate Monitoring#

Do not rely on renewal automation alone. Set up monitoring that alerts before certificates expire.

With Prometheus blackbox_exporter:

# blackbox.yml
modules:
  tls_check:
    prober: tcp
    tls: true
    tls_config:
      insecure_skip_verify: false

# Prometheus alerting rule
- alert: TLSCertExpiringSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 14
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "TLS certificate expires in less than 14 days"

With cert-manager, monitor the certmanager_certificate_expiration_timestamp_seconds metric and alert if any certificate is within 14 days of expiry without a successful renewal.

Common Gotchas#

Missing intermediate certificate: the single most common TLS issue. The server sends only the leaf certificate. Browsers fetch the missing intermediate from the Authority Information Access (AIA) extension in the certificate, so the site works in Chrome. But curl, Python requests, Go HTTP clients, and many mobile apps do not fetch intermediates – they fail with certificate verify failed. Always send the full chain.

Let’s Encrypt rate limits: production Let’s Encrypt has a limit of 50 certificates per registered domain per week. During development and testing, use the staging server (https://acme-staging-v02.api.letsencrypt.org/directory) which has much higher limits. Staging certificates are not trusted by browsers but are functionally identical for testing cert-manager configurations. Also note the 5 duplicate certificates per week limit – requesting the exact same set of hostnames more than 5 times in a week is blocked.