Scenario: Migrating Workloads Between Kubernetes Clusters#

You are helping when someone says: “we need to move workloads from cluster A to cluster B.” The reasons vary – Kubernetes version upgrade, cloud provider migration, region change, architecture consolidation, or moving from self-managed to a managed service. The complexity ranges from trivial (stateless services with GitOps) to significant (stateful workloads with zero-downtime requirements).

The core risk in any cluster migration is data loss for stateful workloads and downtime during the traffic cutover. Every decision in this plan aims to minimize both.

Step 1 – Understand the Scope#

Before touching any infrastructure, answer these questions. The answers determine which migration path to take.

What is migrating?

# Full inventory of the source cluster
kubectl get namespaces
kubectl get deployments --all-namespaces
kubectl get statefulsets --all-namespaces
kubectl get daemonsets --all-namespaces
kubectl get cronjobs --all-namespaces

# Count total workloads
echo "Deployments: $(kubectl get deployments -A --no-headers | wc -l)"
echo "StatefulSets: $(kubectl get statefulsets -A --no-headers | wc -l)"
echo "CronJobs: $(kubectl get cronjobs -A --no-headers | wc -l)"
echo "Services: $(kubectl get svc -A --no-headers | wc -l)"
echo "PVCs: $(kubectl get pvc -A --no-headers | wc -l)"

Source and destination:

  • Same cloud provider, same region: simplest (PV snapshots work, same IAM, same networking)
  • Same cloud, different region: PV snapshots do not work cross-region, need data replication
  • Cross-cloud (AWS to GCP, etc.): no shared infrastructure, everything needs to be recreated
  • On-prem to cloud: networking changes, storage backend changes, potentially different CNI

Stateful vs stateless: Stateless workloads are straightforward – redeploy from manifests. Stateful workloads (databases, message queues, any pod with a PVC) require data migration, which is the hard part.

Acceptable downtime:

  • Zero downtime: requires running both clusters simultaneously with traffic splitting and data replication
  • Maintenance window (1-4 hours): simpler, allows a clean cutover
  • Gradual migration (days/weeks): migrate services one at a time, both clusters run in parallel

Step 2 – Inventory the Source Cluster#

Create a complete inventory. Missing a CRD or a cluster-scoped resource during migration causes failures in the destination that are painful to debug.

Export Kubernetes Manifests#

# Export all resources from application namespaces
# Do NOT export kube-system or other infrastructure namespaces -- recreate those from scratch
NAMESPACES="production staging payments analytics"

for ns in $NAMESPACES; do
  mkdir -p cluster-export/$ns
  for resource in deployments statefulsets services configmaps secrets ingresses \
    horizontalpodautoscalers poddisruptionbudgets networkpolicies serviceaccounts \
    roles rolebindings; do
    kubectl get $resource -n $ns -o yaml > cluster-export/$ns/$resource.yaml 2>/dev/null
  done
done

Identify CRDs and Custom Resources#

# List all CRDs -- these must be installed on the destination BEFORE migrating workloads
kubectl get crd -o custom-columns=NAME:.metadata.name,GROUP:.spec.group

# Export CRD definitions
kubectl get crd -o yaml > cluster-export/crds.yaml

# Common CRDs that need explicit installation:
# - cert-manager.io (Certificate, ClusterIssuer)
# - argoproj.io (Application, AppProject)
# - monitoring.coreos.com (ServiceMonitor, PrometheusRule)
# - keda.sh (ScaledObject)
# - networking.istio.io (VirtualService, Gateway)

Catalog External Dependencies#

# Identify external endpoints referenced in ConfigMaps and Secrets
kubectl get configmaps -A -o json | jq -r '
  .items[] |
  select(.metadata.namespace != "kube-system") |
  "\(.metadata.namespace)/\(.metadata.name): " +
  ([.data // {} | to_entries[] | .value | scan("(?:https?://|jdbc:|amqp://|redis://)[^\\s\"]+")]
  | unique | join(", "))
' | grep -v ': $'

# List PVCs with their storage classes and sizes
kubectl get pvc -A -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
SIZE:.spec.resources.requests.storage,\
CLASS:.spec.storageClassName,\
STATUS:.status.phase

Document Network Configuration#

# Ingress rules (these define your external traffic routing)
kubectl get ingress -A -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
HOSTS:.spec.rules[*].host

# DNS records pointing to the cluster (check your DNS provider)
# These will need to be updated during cutover

# TLS certificates and their issuers
kubectl get certificates -A 2>/dev/null
kubectl get clusterissuers 2>/dev/null

Step 3 – Prepare the Destination Cluster#

The destination cluster must be ready to receive workloads before you begin migrating. The order matters: infrastructure components must be installed before application workloads that depend on them.

Foundation#

# Verify destination cluster health
kubectl --context=destination get nodes
kubectl --context=destination get pods -n kube-system

# Create namespaces
for ns in $NAMESPACES; do
  kubectl --context=destination create namespace $ns --dry-run=client -o yaml | \
    kubectl --context=destination apply -f -
done

# Apply ResourceQuotas and LimitRanges to match source
kubectl get resourcequota -A -o yaml | kubectl --context=destination apply -f -
kubectl get limitrange -A -o yaml | kubectl --context=destination apply -f -

Install Operators and CRDs First#

Install these before any workloads. If a CRD is missing when you apply a custom resource, the apply fails silently or with a confusing error.

# cert-manager
helm install cert-manager jetstack/cert-manager \
  --kube-context=destination \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

# ArgoCD (if using GitOps)
helm install argocd argo/argo-cd \
  --kube-context=destination \
  --namespace argocd --create-namespace

# Prometheus stack
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --kube-context=destination \
  --namespace monitoring --create-namespace

# Install any other CRDs from the source inventory
kubectl --context=destination apply -f cluster-export/crds.yaml

Configure Storage and Networking#

# Set up storage classes matching the source
# If cross-cloud, map source storage classes to destination equivalents
# Example: AWS gp3 -> GCP pd-ssd
kubectl --context=destination apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

# Install ingress controller
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --kube-context=destination \
  --namespace ingress-system --create-namespace \
  --set controller.replicaCount=2

Configure RBAC#

# Copy RBAC from source to destination
kubectl get clusterroles -o yaml | \
  grep -v "kubernetes.io\|system:" | \
  kubectl --context=destination apply -f -

kubectl get clusterrolebindings -o yaml | \
  grep -v "kubernetes.io\|system:" | \
  kubectl --context=destination apply -f -

Step 4 – Migrate Stateless Workloads#

Stateless workloads have no persistent data. They can be redeployed from source (manifests, Helm charts, or GitOps).

Option A: GitOps (Preferred)#

If you use ArgoCD or Flux, point them at both clusters. The same git repository deploys to both.

# ArgoCD: add the destination cluster
argocd cluster add destination-context --name destination

# Create Applications targeting the destination cluster
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-destination
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/k8s-manifests.git
    targetRevision: main
    path: production
  destination:
    name: destination
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Option B: Velero Backup/Restore#

Velero can back up all Kubernetes resources from the source and restore them to the destination.

# On the source cluster: create a backup
velero backup create migration-backup \
  --include-namespaces production,staging,payments \
  --exclude-resources persistentvolumeclaims,persistentvolumes \
  --default-volumes-to-fs-backup=false

# Wait for completion
velero backup describe migration-backup --details

# On the destination cluster: install Velero pointing to the same bucket
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket velero-migration \
  --backup-location-config region=us-east-1 \
  --secret-file ./credentials-velero

# Verify the backup is visible from the destination
velero backup get

# Restore
velero restore create --from-backup migration-backup

# Check restore status
velero restore describe <restore-name> --details

Option C: Manual Apply#

# Apply exported manifests to the destination
for ns in $NAMESPACES; do
  for file in cluster-export/$ns/*.yaml; do
    kubectl --context=destination apply -f "$file" 2>&1 | grep -v "unchanged"
  done
done

Verify Stateless Migration#

# Compare workload counts between source and destination
echo "=== Source ==="
kubectl get deployments -A --no-headers | wc -l
echo "=== Destination ==="
kubectl --context=destination get deployments -A --no-headers | wc -l

# Check all pods are running on destination
kubectl --context=destination get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

# Verify services are created
kubectl --context=destination get svc -A

Step 5 – Migrate Stateful Workloads#

This is the hard part. Data must be moved without loss, and ideally without downtime.

Option A: Velero with PV Snapshots (Same Cloud Provider Only)#

# Source cluster: backup including volumes
velero backup create stateful-migration \
  --include-namespaces payments \
  --default-volumes-to-fs-backup

# This uses Kopia to copy PV data to the object storage bucket
# Cross-cloud: data goes through S3/GCS, accessible from either side

# Destination cluster: restore
velero restore create --from-backup stateful-migration

# Verify PVCs are created and bound
kubectl --context=destination get pvc -n payments

Caveat: Velero file-level backup is a point-in-time snapshot. Any data written after the backup starts and before the cutover is lost unless you use continuous replication.

Option B: Application-Level Dump/Restore#

For databases, application-level tools give the most control and work cross-cloud.

# PostgreSQL
# On source: dump the database
kubectl exec -n payments deploy/postgres -- \
  pg_dump -U postgres -Fc payments_db > payments_db.dump

# Copy the dump to the destination
kubectl cp payments_db.dump payments/postgres-0:/tmp/payments_db.dump \
  --context=destination

# On destination: restore
kubectl exec -n payments deploy/postgres --context=destination -- \
  pg_restore -U postgres -d payments_db -c /tmp/payments_db.dump

# Verify row counts
kubectl exec -n payments deploy/postgres -- \
  psql -U postgres -d payments_db -c "SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
kubectl exec -n payments deploy/postgres --context=destination -- \
  psql -U postgres -d payments_db -c "SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
# MySQL
kubectl exec -n payments deploy/mysql -- \
  mysqldump -u root -p"$MYSQL_ROOT_PASSWORD" --all-databases --single-transaction > mysql_dump.sql

kubectl cp mysql_dump.sql payments/mysql-0:/tmp/mysql_dump.sql --context=destination

kubectl exec -n payments deploy/mysql --context=destination -- \
  mysql -u root -p"$MYSQL_ROOT_PASSWORD" < /tmp/mysql_dump.sql

Option C: Continuous Replication (Zero Downtime)#

For zero-downtime migration, set up replication from source to destination. The destination follows the source in real-time, and you cut over when they are in sync.

# PostgreSQL logical replication
# Source: enable logical replication in postgresql.conf
# wal_level = logical
# max_replication_slots = 4

# Source: create publication
kubectl exec -n payments deploy/postgres -- \
  psql -U postgres -d payments_db -c "CREATE PUBLICATION migration_pub FOR ALL TABLES;"

# Destination: create subscription (after initial schema sync)
kubectl exec -n payments deploy/postgres --context=destination -- \
  psql -U postgres -d payments_db -c "
    CREATE SUBSCRIPTION migration_sub
    CONNECTION 'host=source-db.internal port=5432 dbname=payments_db user=replicator password=secret'
    PUBLICATION migration_pub;
  "

# Monitor replication lag
kubectl exec -n payments deploy/postgres -- \
  psql -U postgres -c "SELECT slot_name, confirmed_flush_lsn FROM pg_replication_slots;"

Once replication lag is zero, you are ready for cutover.

Step 6 – Traffic Cutover#

This is the moment of truth. Traffic switches from the old cluster to the new one.

Option A: DNS Cutover#

The simplest approach. Change DNS records to point to the new cluster’s load balancer.

# Get the new cluster's load balancer IP/hostname
kubectl --context=destination get svc -n ingress-system ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

# Update DNS records (example using AWS Route53)
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "new-lb-xxxxx.elb.amazonaws.com"}]
      }
    }]
  }'

Important: Lower the DNS TTL to 60 seconds several days before migration. If your TTL is 3600 (1 hour), some clients will continue hitting the old cluster for up to an hour after the DNS change. Set it low in advance:

# Lower TTL days before migration
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "old-lb-xxxxx.elb.amazonaws.com"}]
      }
    }]
  }'

Option B: Weighted Routing (Gradual Cutover)#

Split traffic between old and new clusters to validate before committing fully.

# AWS Route53 weighted routing
# Start: 95% old, 5% new
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "old-cluster",
          "Weight": 95,
          "TTL": 60,
          "ResourceRecords": [{"Value": "old-lb-xxxxx.elb.amazonaws.com"}]
        }
      },
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "new-cluster",
          "Weight": 5,
          "TTL": 60,
          "ResourceRecords": [{"Value": "new-lb-xxxxx.elb.amazonaws.com"}]
        }
      }
    ]
  }'

# Monitor error rates on the new cluster
# If stable, increase: 50/50, then 10/90, then 0/100
# Each step: wait 15-30 minutes, check error rates and latency

Option C: External Load Balancer Backend Swap#

If you have an external load balancer (AWS ALB, GCP Cloud Load Balancer, Cloudflare) in front of the cluster, swap the backend targets.

# AWS: modify target group targets
# Remove old cluster nodes, add new cluster nodes
aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/my-app/xxxx \
  --targets Id=new-node-1,Port=30080 Id=new-node-2,Port=30080

aws elbv2 deregister-targets \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/my-app/xxxx \
  --targets Id=old-node-1,Port=30080 Id=old-node-2,Port=30080

Step 7 – Validation and Cleanup#

Immediate Validation (First Hour)#

# Verify traffic is flowing to the new cluster
kubectl --context=destination logs deployment/my-app -n production --tail=10
# You should see request logs

# Check error rates on the new cluster
kubectl --context=destination get events -n production --sort-by='.lastTimestamp' | head -20

# Run smoke tests against the new cluster
curl -s https://app.example.com/health | jq .
curl -s https://app.example.com/api/status | jq .

# Verify data integrity for stateful workloads
# Compare row counts, checksums, or application-level health checks
kubectl exec -n payments deploy/postgres --context=destination -- \
  psql -U postgres -d payments_db -c "SELECT COUNT(*) FROM orders;"

Extended Monitoring (24-48 Hours)#

Keep both clusters running for at least 24-48 hours after cutover. The old cluster serves as a fallback.

# On the old cluster: scale down application pods but keep infrastructure running
kubectl scale deployment --all -n production --replicas=0
# Do NOT delete the old cluster yet

# Monitor the new cluster for:
# - Error rate trends (compare to pre-migration baseline)
# - Latency trends (should be comparable or better)
# - Pod restarts (should be zero or near zero)
# - CronJob execution (verify scheduled jobs run successfully on the new cluster)
kubectl --context=destination get cronjobs -A
kubectl --context=destination get jobs -A --sort-by='.metadata.creationTimestamp' | tail -10

Cleanup (After Validation Period)#

# If replication was used, stop it
kubectl exec -n payments deploy/postgres --context=destination -- \
  psql -U postgres -d payments_db -c "DROP SUBSCRIPTION migration_sub;"

# Remove weighted routing (set new cluster to 100%)
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "DELETE",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "CNAME",
        "SetIdentifier": "old-cluster",
        "Weight": 0,
        "TTL": 60,
        "ResourceRecords": [{"Value": "old-lb-xxxxx.elb.amazonaws.com"}]
      }
    }]
  }'

# Raise DNS TTL back to normal (3600 or whatever your standard is)

# Decommission old cluster resources
# Delete old load balancers
# Delete old PVs and PVCs
# Remove old node pools
# Delete old cluster (only after extended validation!)

# Update documentation
# - Runbooks: update cluster endpoints, kubectl contexts
# - On-call docs: update escalation paths if cluster context changed
# - CI/CD: update deployment targets to point to new cluster
# - Monitoring: verify dashboards reference the new cluster

Migration Checklist Summary#

Step Gate Criteria
Scope definition Know what is moving, where, and acceptable downtime
Source inventory All namespaces, CRDs, PVCs, external dependencies documented
Destination prep All CRDs installed, storage classes configured, ingress working
Stateless migration All deployments running, services reachable on destination
Stateful migration Data transferred, row counts match, replication lag zero
Traffic cutover DNS TTL lowered, weighted routing or DNS switch executed
Validation Error rates stable, latency normal, smoke tests passing
Cleanup Old cluster decommissioned, documentation updated

The most common migration failures are: missing CRDs on the destination (install operators before workloads), storage class mismatches (map source classes to destination equivalents), and premature old cluster decommission (keep it running for at least 48 hours as a fallback). Address these three risks explicitly and the migration is mechanical.