Certificate Basics#

A TLS certificate binds a public key to a domain name. The certificate is signed by a Certificate Authority (CA) that browsers and operating systems trust. The chain goes: your certificate, signed by an intermediate CA, signed by a root CA. All three must be present and valid for a client to trust the connection.

Self-Signed Certificates for Development#

For local development and testing, generate a self-signed certificate. Clients will not trust it by default, but you can add it to your local trust store.

Generate a private key and self-signed certificate in one command:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem \
  -days 365 -nodes \
  -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,DNS:*.local.dev,IP:127.0.0.1"

The -nodes flag means no passphrase on the private key (acceptable for development, never for production). The subjectAltName extension is required by modern browsers – the Common Name (CN) field alone is no longer sufficient.

For a local CA that signs multiple dev certificates:

# Create CA key and certificate
openssl req -x509 -newkey rsa:4096 -keyout ca-key.pem -out ca-cert.pem \
  -days 3650 -nodes -subj "/CN=Local Dev CA"

# Generate a server key and CSR
openssl req -newkey rsa:2048 -keyout server-key.pem -out server.csr \
  -nodes -subj "/CN=myapp.local.dev"

# Sign the CSR with your CA
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -out server-cert.pem -days 365 \
  -extfile <(printf "subjectAltName=DNS:myapp.local.dev,DNS:api.local.dev")

Add ca-cert.pem to your OS trust store once, and all certificates signed by it will be trusted. On macOS: sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ca-cert.pem. On Ubuntu: copy to /usr/local/share/ca-certificates/ and run update-ca-certificates.

Generating a CSR for a Real Certificate#

When purchasing a certificate or using an internal CA, submit a Certificate Signing Request:

openssl req -newkey rsa:2048 -keyout domain-key.pem -out domain.csr \
  -nodes -subj "/C=US/ST=California/L=San Francisco/O=MyOrg/CN=example.com"

Inspect before submitting with openssl req -in domain.csr -text -noout. Keep the private key safe – if compromised, the certificate must be revoked.

Let’s Encrypt with Certbot#

Let’s Encrypt provides free, automated certificates trusted by all browsers. Certbot is the standard client.

Standalone mode (certbot runs its own web server on port 80):

certbot certonly --standalone -d example.com -d www.example.com

Webroot mode (certbot writes a challenge file to your existing web server’s document root):

certbot certonly --webroot -w /var/www/html -d example.com

DNS challenge (for wildcard certificates or when port 80 is not reachable):

certbot certonly --manual --preferred-challenges dns -d "*.example.com"

Certbot stores certificates in /etc/letsencrypt/live/example.com/. The key files are privkey.pem (private key), fullchain.pem (certificate + intermediate), and chain.pem (intermediate only). Always configure your server with fullchain.pem, not cert.pem, so clients receive the full chain.

Automate renewal with a cron job or systemd timer. Certbot installs a systemd timer by default on most distributions:

systemctl status certbot.timer
certbot renew --dry-run    # test renewal

cert-manager in Kubernetes#

cert-manager automates certificate issuance and renewal inside a Kubernetes cluster. Install it with Helm:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

Create a ClusterIssuer for Let’s Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx

Then annotate your Ingress to request a certificate automatically:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 8080

cert-manager creates a Certificate resource, performs the ACME challenge, stores the cert in the myapp-tls Secret, and renews it before expiry. Check status with kubectl get certificates -A and kubectl describe certificate myapp-tls.

Debugging TLS Issues#

openssl s_client is the primary debugging tool:

# Show certificate chain (use -servername for SNI)
openssl s_client -connect example.com:443 -servername example.com </dev/null

# Check expiration dates
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

# Inspect a certificate file
openssl x509 -in cert.pem -text -noout

# Verify cert matches private key (modulus must match)
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5

Common Errors and Fixes#

Certificate expired. Check the notAfter date with openssl x509 -enddate -noout -in cert.pem. Renew the certificate and reload the server.

Hostname mismatch. The domain does not match the certificate’s SAN list. Inspect with openssl x509 -noout -text -in cert.pem | grep -A1 "Subject Alternative Name".

Untrusted CA / incomplete chain. The server sends the leaf certificate but not the intermediate. Use fullchain.pem and look for “Verify return code: 21” in openssl s_client output.

TLS termination confusion. If TLS terminates at a load balancer, backends see HTTP. The app must check X-Forwarded-Proto to determine the original protocol.

Permission denied on key file. Private keys should be 600 or 640, owned by root and readable only by the service user.