EKS Networking and Load Balancing#

EKS networking differs fundamentally from generic Kubernetes networking. Pods get real VPC IP addresses, load balancers are AWS-native resources, and networking decisions have direct cost and IP capacity implications.

VPC CNI: How Pod Networking Works#

The AWS VPC CNI plugin assigns each pod an IP address from your VPC CIDR. Unlike overlay networks (Calico, Flannel), pods are directly routable within the VPC. This means security groups, NACLs, and VPC flow logs all work with pod traffic natively.

Each EC2 instance has a limit on how many ENIs and IPs per ENI it supports. An m6i.large supports 3 ENIs with 10 IPs each, giving you 29 pod IPs per node (minus one IP per ENI for the node itself). Check limits with:

aws ec2 describe-instance-types --instance-types m6i.large \
  --query "InstanceTypes[0].NetworkInfo.{ENIs:MaximumNetworkInterfaces,IPv4PerENI:Ipv4AddressesPerInterface}"

IP Exhaustion#

The most common VPC CNI problem is running out of IPs. The CNI pre-allocates IPs by keeping a warm pool. In a /24 subnet (254 usable IPs), you can exhaust addresses quickly with a few nodes.

Solutions:

1. Secondary CIDR blocks. Add a 100.64.0.0/16 CIDR to your VPC and configure the CNI to use it for pods. Node primary IPs come from the original CIDR; pod IPs come from the secondary range.

# Enable custom networking
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

# Then create ENIConfig resources per AZ pointing to the secondary subnets

2. Prefix delegation. Instead of assigning individual IPs, the CNI assigns /28 prefixes (16 IPs) to each ENI slot. This dramatically increases pod density. An m6i.large goes from 29 pods to over 100.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Enable prefix delegation on new clusters. Retrofitting it onto existing clusters requires rolling all nodes.

Security Groups for Pods#

By default, all pods share the node’s security groups. Security Groups for Pods lets you assign specific security groups to individual pods, useful for restricting database access to only the pods that need it.

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: db-access
  namespace: production
spec:
  podSelector:
    matchLabels:
      db-access: "true"
  securityGroups:
    groupIds:
      - sg-0123456789abcdef0

This requires the VPC CNI to be configured with ENABLE_POD_ENI=true and only works on Nitro-based instances.

AWS Load Balancer Controller#

The AWS Load Balancer Controller creates and manages ALBs (for Ingress) and NLBs (for Service type LoadBalancer). It replaces the legacy in-tree cloud provider load balancer.

Install it:

helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=true \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/aws-lbc-role

The controller needs an IAM role with permissions to create/manage ALBs, NLBs, target groups, and security groups. AWS publishes the required IAM policy in their documentation.

NLB for Service Type LoadBalancer#

Annotate a Service to get a Network Load Balancer:

apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: production
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-aaa,subnet-bbb"
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - port: 443
      targetPort: 8080
      protocol: TCP

Key annotations:

  • aws-load-balancer-type: "external" – tells the AWS LB Controller (not the legacy cloud provider) to handle this.
  • aws-load-balancer-nlb-target-type: "ip" – routes directly to pod IPs. Use "instance" to route to node ports instead. IP mode is faster (skips a hop) and works with Fargate.
  • aws-load-balancer-scheme: "internet-facing" – public NLB. Use "internal" for private.

For TLS termination on the NLB:

annotations:
  service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:123456789012:certificate/abc-123"
  service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

ALB for Ingress#

The controller creates an ALB when it sees an Ingress with ingressClassName: alb:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: production
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-east-1:123456789012:certificate/abc-123"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/group.name: shared-alb
spec:
  ingressClassName: alb
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 80

The group.name annotation merges multiple Ingress resources into a single ALB. Without it, every Ingress creates its own ALB (~$16/month each). You can also configure health check interval, thresholds, and timeout via additional alb.ingress.kubernetes.io/healthcheck-* annotations.

Instance Mode vs IP Mode#

IP mode (target-type: ip): traffic goes directly to pod IPs. Lower latency, works with Fargate, skips the NodePort hop. Instance mode (target-type: instance): traffic goes to node IPs on a NodePort. Default to IP mode for new setups.

ExternalDNS with Route53#

ExternalDNS watches Services and Ingresses and creates DNS records in Route53 automatically.

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns
helm install external-dns external-dns/external-dns \
  -n kube-system \
  --set provider.name=aws \
  --set policy=upsert-only \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/external-dns-role

The IAM role needs route53:ChangeResourceRecordSets on the hosted zone and route53:ListHostedZones / route53:ListResourceRecordSets.

Annotate your Ingress or Service:

annotations:
  external-dns.alpha.kubernetes.io/hostname: "app.example.com"
  external-dns.alpha.kubernetes.io/ttl: "300"

Use policy: upsert-only in production to prevent ExternalDNS from deleting records it did not create. Switch to policy: sync only if ExternalDNS exclusively owns the hosted zone.

Backward EKS IAM and Security EKS Setup and Configuration Forward