Debugging ArgoCD#

Most ArgoCD problems fall into predictable categories: sync stuck in a bad state, resources showing OutOfSync when they should not be, health checks reporting wrong status, RBAC blocking operations, or repository connections failing. Here is how to diagnose and fix each one.

Application Stuck in Progressing#

An application stuck in Progressing means ArgoCD is waiting for a resource to become healthy and it never does. The most common causes:

# See which specific resource is not healthy
argocd app get my-app --show-operation

# Get detailed resource status
argocd app resources my-app --output wide

Deployment not rolling out: The new ReplicaSet cannot schedule pods. Check the pods directly:

kubectl get pods -n my-app -l app=my-app
kubectl describe pod <stuck-pod> -n my-app

Common causes: image pull failures (wrong tag, missing registry credentials), insufficient resources (no node has enough CPU or memory), PVC not bound, init containers failing.

Service with type LoadBalancer: ArgoCD waits for the external IP to be assigned. On bare-metal or minikube without a LoadBalancer provider, this never happens. Either install MetalLB or change the service type.

Custom resources without health checks: ArgoCD does not know how to check health for arbitrary CRDs. It defaults to treating them as Progressing until you configure a custom health check (see below).

OutOfSync When It Should Not Be#

The app shows OutOfSync even though you have not changed anything in Git. This is one of the most frustrating ArgoCD issues.

# See exactly what ArgoCD thinks is different
argocd app diff my-app

Server-side defaulting: Kubernetes mutates resources after they are applied. Fields like spec.template.spec.terminationGracePeriodSeconds default to 30, spec.revisionHistoryLimit defaults to 10, and so on. If your manifest does not set these fields, the live state has them but the desired state does not. ArgoCD sees a diff.

Fixes:

  1. Explicitly set the defaulted fields in your manifests.
  2. Use ServerSideApply=true sync option, which handles defaulting more gracefully.
  3. Add a managedFields ignore difference:
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/revisionHistoryLimit

Mutating webhooks: An admission webhook modifies resources after ArgoCD applies them. Sidecar injectors (like Istio) add containers and volumes. Resource mutators add labels or annotations.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.spec.containers[] | select(.name == "istio-proxy")
        - .spec.template.metadata.annotations."sidecar.istio.io/status"

Resource tracking mode conflicts: ArgoCD tracks which resources it manages using either labels or annotations. If you switch modes or have leftover tracking metadata, ArgoCD gets confused. See the Resource Tracking section below.

Sync Failed Errors#

# View the sync operation result
argocd app get my-app

# Check logs from the application controller
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100

# Check repo server logs for manifest generation errors
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=100

“the object has been modified”: A conflict between ArgoCD and another controller modifying the same resource. Use ServerSideApply=true to resolve field ownership conflicts.

“immutable field cannot be updated”: Certain fields on Jobs, PVCs, and StatefulSets cannot be changed after creation. Options: use Replace=true sync option (deletes and recreates the resource), or delete the resource manually and let ArgoCD recreate it.

“exceeded quota”: The namespace has a ResourceQuota and the new resources exceed it. Check quotas with kubectl describe resourcequota -n <namespace>.

Custom Health Checks#

ArgoCD has built-in health checks for standard Kubernetes resources. For CRDs or resources with non-standard health semantics, write custom checks in Lua.

Add custom health checks to the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.certmanager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate to be issued"
    return hs

The Lua script receives the resource as obj and must return a table with status (Healthy, Progressing, Degraded, Suspended, Missing) and message.

Resource Tracking: Label vs Annotation#

ArgoCD tracks which resources it manages using one of three modes configured in argocd-cm:

data:
  application.resourceTrackingMethod: annotation  # or label, or annotation+label
  • label (legacy default): Adds app.kubernetes.io/instance: <app-name> label. Conflicts with Helm, which uses the same label. Causes incorrect ownership detection if multiple tools set this label.
  • annotation: Uses argocd.argoproj.io/tracking-id annotation. Does not conflict with other tools. This is the recommended setting.
  • annotation+label: Sets both. Useful during migration from label to annotation mode.

If you are seeing resources claimed by the wrong application, or ArgoCD refusing to manage resources it should, check your tracking method and look for stale labels or annotations from a previous configuration.

RBAC Permission Errors#

ArgoCD RBAC is defined in the argocd-rbac-cm ConfigMap. When users get permission denied errors:

# Check what policies are loaded
kubectl get cm argocd-rbac-cm -n argocd -o yaml

# Test a specific permission
argocd admin settings rbac validate --policy-file policy.csv

RBAC policy format:

p, role:developer, applications, get, */*, allow
p, role:developer, applications, sync, */*, allow
p, role:developer, applications, create, */*, deny
g, my-team, role:developer

Common RBAC issues:

  • Missing group binding: The SSO group is not mapped to an ArgoCD role. Add a g line mapping the group to a role.
  • Project scope mismatch: The policy grants access to default/* but the app is in project production. Policies are project/application scoped.
  • Default policy too restrictive: The policy.default key in argocd-rbac-cm controls what unauthenticated or unmapped users can do. Set it to role:readonly for a reasonable default, or empty string to deny everything.

Repository Connection Failures#

# Test repo connectivity
argocd repo list
argocd repo get https://github.com/myorg/k8s-manifests.git

# Check repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=50

SSH key issues: The repo was added with an SSH key that has since been rotated. Update it:

argocd repo add git@github.com:myorg/repo.git --ssh-private-key-path ~/.ssh/new_key --upsert

GitHub token expired: Personal access tokens and GitHub App installation tokens expire. Check the secret storing the credential:

kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=repository

Self-signed certificates: For private Git servers with self-signed TLS certs, add the CA certificate:

argocd cert add-tls git.internal.example.com --from /path/to/ca.crt

Or add it to the argocd-tls-certs-cm ConfigMap.

SSO and OIDC Issues#

The two most common SSO failures are redirect URI mismatches and missing group claims. The callback URL in your OIDC provider must be https://<argocd-url>/auth/callback, and the url field in argocd-cm must match exactly what users see in their browser. For RBAC to work, the ID token must include a groups claim – configure requestedScopes and requestedIDTokenClaims in the OIDC config if your provider does not include groups by default. Debug SSO issues by checking Dex logs:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-dex-server --tail=100

Essential Debugging Commands#

# Full app status with resource details
argocd app get my-app

# Diff between desired (Git) and live (cluster) state
argocd app diff my-app

# Logs from a specific resource managed by ArgoCD
argocd app logs my-app --resource-name my-deployment --container my-container

# Force ArgoCD to re-read manifests from Git
argocd app get my-app --hard-refresh

# List all resources and their sync/health status
argocd app resources my-app

# View the rendered manifests ArgoCD would apply
argocd app manifests my-app --source git
argocd app manifests my-app --source live