Troubleshooting Pods

The Troubleshooting Decision Tree

Always follow this workflow. Do not skip steps.

kubectl get pods
       │
       ├── CrashLoopBackOff ──► kubectl logs <pod> --previous
       │                         kubectl describe pod <pod> (check Exit Code, Last State)
       │
       ├── OOMKilled ──────────► kubectl describe pod <pod> (look for Exit Code: 137)
       │                         Increase resources.limits.memory
       │
       ├── ImagePullBackOff ───► kubectl describe pod <pod> (Events: Failed to pull image)
       │                         Check image name/tag spelling
       │                         Check imagePullSecrets for private registries
       │
       ├── Pending ────────────► kubectl describe pod <pod> (Events: FailedScheduling)
       │                         kubectl get nodes (are nodes Ready?)
       │                         kubectl describe node <node> (check Allocatable vs Requests)
       │                         Check nodeSelector, tolerations, affinity rules
       │
       ├── Init:0/1 ───────────► kubectl logs <pod> -c <init-container-name>
       │                         kubectl describe pod <pod> (Init Containers section)
       │
       └── Running but unhealthy► kubectl exec -it <pod> -- /bin/sh
                                   kubectl logs <pod> -f
                                   kubectl describe pod <pod> (check probe failures)

CrashLoopBackOff

The most common failure. Kubernetes restarts a crashing container with exponential backoff:

Restart 1:   wait 10s
Restart 2:   wait 20s
Restart 3:   wait 40s
Restart 4:   wait 80s
Restart 5:   wait 160s
Restart 6+:  wait 300s (5 min cap)

This means after several restarts, the container is only UP for a few seconds every 5 minutes. Running kubectl logs during the backoff window shows an empty or stale log. Use --previous to see the terminated container's logs:

kubectl logs <pod> --previous
kubectl logs <pod> -c <container> --previous   # multi-container pods

The kubectl describe output tells you the exit code:

Last State:  Terminated
  Reason:    Error
  Exit Code: 1           ← application crashed
  Exit Code: 137          ← OOMKilled (128 + SIGKILL signal 9)
  Exit Code: 126          ← permission denied (can't execute command)
  Exit Code: 127          ← command not found (bad entrypoint)

OOMKilled

When a container exceeds its resources.limits.memory, the Linux kernel OOM killer sends SIGKILL (signal 9). Exit code = 128 + 9 = 137.

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137
  Started:   Mon, 05 Jun 2026 10:01:00 +0000
  Finished:  Mon, 05 Jun 2026 10:01:03 +0000

Fix: increase resources.limits.memory or profile the application for a memory leak.

ImagePullBackOff

Kubernetes tries to pull the image, fails (ErrImagePull), then retries with backoff (ImagePullBackOff). Common causes:

1. Wrong image name or tag — typo in the image field
2. Private registry — no imagePullSecrets configured
3. Rate limiting — Docker Hub pull limit (unauthenticated: 100 pulls/6h)

Events:
  Warning  Failed     3s    kubelet  Failed to pull image "nginx:does-not-exist":
           rpc error: code = NotFound desc = failed to pull and unpack image
           "docker.io/library/nginx:does-not-exist": not found
  Warning  Failed     3s    kubelet  Error: ErrImagePull
  Warning  BackOff    2s    kubelet  Back-off pulling image "nginx:does-not-exist"

Pending — Scheduler Cannot Place the Pod

The scheduler evaluates every node and rejects them all. The Events section explains why:

Events:
  Warning  FailedScheduling  5s  default-scheduler
    0/2 nodes are available:
    1 Insufficient cpu.
    1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: ""}.

Decode "0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had taint":

• Node 1: not enough CPU
• Node 2: has a control-plane taint and the Pod has no toleration

Other common Pending causes:

• nodeSelector with a label that no node has
• PersistentVolumeClaim in Pending state (no matching PV)
• Resource quota exhausted in the namespace

Init Container Failures

Init containers run sequentially before app containers start. If one fails, the Pod status shows Init:0/2 (0 of 2 init containers completed).

# Check init container logs — must specify -c with the init container name
kubectl logs <pod> -c init-db-check

# describe shows each init container's state separately
kubectl describe pod <pod>
# Look for:
# Init Containers:
#   init-db-check:
#     State:    Terminated
#     Reason:   Error
#     Exit Code: 1