kubectl debug and Ephemeral Containers#

Production containers should be minimal. Distroless images, scratch-based Go binaries, and hardened base images strip out shells, package managers, and debugging tools. This is good for security and image size, but it means kubectl exec gives you nothing to work with. Ephemeral containers solve this problem.

The Problem#

A typical distroless container has no shell:

$ kubectl exec -it payments-api-7f8b9c6d4-x2k9m -- /bin/sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory

You cannot install tools, you cannot inspect files, and you cannot run any diagnostic commands. The application is returning 500 errors and you have nothing but logs.

Ephemeral Containers#

Ephemeral containers (GA since Kubernetes v1.25) are temporary containers that you add to a running pod for debugging. They are not defined in the pod spec. They do not restart if they exit. They exist purely for investigation.

The kubectl debug command is the primary interface for creating ephemeral containers. It operates in three distinct modes.

Mode 1: Attach to an Existing Pod#

This is the most common mode. You add a debug container directly to the running pod without restarting it.

kubectl debug -it payments-api-7f8b9c6d4-x2k9m \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -n payments-prod

What happens:

  • An ephemeral container running nicolaka/netshoot is added to the existing pod
  • The --target=payments-api flag shares the process namespace with the payments-api container
  • You get a shell inside the debug container with full networking tools
  • The original pod keeps running – no restart, no downtime

Process namespace sharing#

With --target, the debug container shares the PID namespace of the target container. This means ps aux in the debug container shows the target’s processes:

# Inside the debug container
$ ps aux
PID   USER     COMMAND
1     root     /app/payments-api --config=/etc/config/app.yaml
12    root     sh   # this is you, in the debug container

Filesystem access#

You can reach the target container’s filesystem through /proc/1/root/:

# Read the target container's config file
cat /proc/1/root/etc/config/app.yaml

# Check mounted secrets
ls /proc/1/root/var/run/secrets/kubernetes.io/serviceaccount/

# Inspect the application binary
file /proc/1/root/app/payments-api

This works because PID 1 in the shared namespace is the target container’s main process, and /proc/<pid>/root/ is a symlink to that process’s root filesystem.

Network debugging#

The debug container shares the pod’s network namespace, so all network tests reflect the pod’s actual connectivity:

# DNS resolution
nslookup postgres-primary.database.svc.cluster.local

# HTTP connectivity to another service
curl -v http://order-service.orders.svc.cluster.local:8080/health

# Check what ports the application is listening on
ss -tlnp

# Packet capture (netshoot has tcpdump)
tcpdump -i eth0 -n port 5432

Mode 2: Copy Pod for Debugging#

Sometimes you need to change the pod’s configuration to debug it – different image, different command, different environment variables. Copy mode creates a clone of the pod that you can modify freely.

# Copy the pod, replacing its image with busybox
kubectl debug payments-api-7f8b9c6d4-x2k9m -it \
  --copy-to=debug-payments \
  --set-image=payments-api=busybox \
  --share-processes \
  -n payments-prod

What happens:

  • A new pod named debug-payments is created as a copy of the original
  • The payments-api container image is replaced with busybox
  • --share-processes enables PID namespace sharing between all containers in the copy
  • The original pod is completely untouched

This is useful when:

  • You want to run the application with a different entrypoint to test something
  • You need to add environment variables or change the command
  • The application crashes too fast to attach a debug container
# Copy the pod but override the command to keep it alive
kubectl debug payments-api-7f8b9c6d4-x2k9m -it \
  --copy-to=debug-payments \
  --container=payments-api \
  -- sh -c "sleep infinity"

Now you can exec into debug-payments and manually run the application to see what happens:

kubectl exec -it debug-payments -c payments-api -n payments-prod -- /bin/sh
# Try running the app manually
$ /app/payments-api --config=/etc/config/app.yaml
# See the actual error output in real-time

Clean up the debug copy when you are done:

kubectl delete pod debug-payments -n payments-prod

Mode 3: Debug a Node#

Node-level debugging creates a privileged pod on a specific node with the host filesystem mounted.

kubectl debug node/ip-10-0-1-42.ec2.internal -it --image=busybox

What happens:

  • A privileged pod is created on the target node
  • The host filesystem is mounted at /host
  • You have root access to the node
# Get full host access
chroot /host

# Check kubelet status
systemctl status kubelet
journalctl -u kubelet --since "10 minutes ago"

# Check container runtime
crictl ps
crictl logs <container-id>

# Check disk usage (common cause of evictions)
df -h

# Check system memory
free -m

# Check what pods are actually running on this node
crictl pods

This is invaluable when kubelet is misbehaving, the container runtime has issues, or you need to inspect host-level networking or storage.

Choosing a Debug Image#

The debug image determines what tools you have available. Pick based on what you need to investigate.

Image Size Best For
busybox ~1 MB Basic inspection: sh, ls, cat, wget, vi
alpine ~7 MB When you need a package manager: apk add strace curl
nicolaka/netshoot ~300 MB Full networking toolkit: tcpdump, nslookup, curl, iperf, netstat, dig
ubuntu ~78 MB When you need apt and a familiar environment
Custom Varies Build your own with exactly the tools your team uses

For most debugging sessions, nicolaka/netshoot is the best choice. It has every network tool you are likely to need, plus general-purpose utilities. If you are only checking files or processes, busybox keeps things fast.

Building a custom debug image#

If your team frequently debugs the same type of applications, build a dedicated debug image:

FROM nicolaka/netshoot
RUN apk add --no-cache \
    strace \
    gdb \
    postgresql-client \
    redis
COPY custom-scripts/ /usr/local/bin/

Practical Example: Debugging a Distroless Go Service#

Scenario: payments-api is a distroless Go binary returning HTTP 500 errors. Logs show connection refused when reaching the database, but the database pod is running and other services connect to it successfully.

Step 1: Attach a network debug container.

kubectl debug -it payments-api-7f8b9c6d4-x2k9m \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -n payments-prod

Step 2: Test DNS resolution from inside the pod.

$ nslookup postgres-primary.database.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10#53

Name:   postgres-primary.database.svc.cluster.local
Address: 10.108.42.15

DNS works. The service resolves.

Step 3: Test TCP connectivity to the database port.

$ curl -v telnet://postgres-primary.database.svc.cluster.local:5432
* Trying 10.108.42.15:5432...
* connect to 10.108.42.15 port 5432 failed: Connection refused

Connection refused. The service IP resolves but nothing is listening.

Step 4: Check if the service has endpoints.

# From a different terminal
kubectl get endpointslices -n database -l kubernetes.io/service-name=postgres-primary

Empty endpoint slices. The service has no backing pods. The database pod is running, but its readiness probe is failing, so it is not added to the endpoint slice. Fix the readiness probe or the underlying database issue, and the payments service will recover.

Step 5: Check the app’s configuration.

# Still inside the debug container
cat /proc/1/root/etc/config/app.yaml

Confirm the connection string matches the service name you tested.

Limitations#

Ephemeral containers have restrictions you should understand before relying on them:

  • Cannot be removed. Once added, an ephemeral container stays in the pod spec until the pod terminates. It does not consume resources after it exits, but it shows up in kubectl describe pod.
  • Cannot mount new volumes. The debug container can only access volumes already mounted in the pod. You cannot add a new volume mount.
  • Cannot change resource limits. The debug container inherits the pod’s cgroup but cannot modify resource allocations.
  • RBAC required. Creating ephemeral containers requires the pods/ephemeralcontainers subresource permission. Add this to your debug RBAC role:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

Security Considerations#

Debug containers with --target can see the target container’s process memory, filesystem, and environment variables. This means a debug session can read database passwords from environment variables, TLS private keys from mounted secrets, and application memory containing sensitive data.

Restrict who can create ephemeral containers. In production, this should be limited to on-call engineers and require audit logging. Some organizations require a break-glass process to get debug access to production namespaces.

Node debugging is even more sensitive – chroot /host gives full root access to the node, including access to every container’s filesystem and the kubelet’s credentials. Treat kubectl debug node/ as equivalent to SSH root access.