Troubleshooting Nodes

Diagnose NotReady nodes by reading kubelet logs, inspecting node conditions, and safely cordoning and draining problem nodes.

advanced75 minRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: A node goes NotReady. You have 10 pods on it. Without any intervention, what happens to those pods? How long before Kubernetes tries to reschedule them? What is the mechanism — and what do node taints have to do with it?

Node Conditions — The Diagnostic Map

kubectl describe node <node-name>
# Look for the Conditions section:

Conditions:
  Type              Status  Reason                 Message
  ────────────────  ──────  ─────────────────────  ──────────────────────────────────
  MemoryPressure    False   KubeletHasSufficientMemory  kubelet has sufficient memory
  DiskPressure      False   KubeletHasNoDiskPressure    kubelet has no disk pressure
  PIDPressure       False   KubeletHasSufficientPID     kubelet has sufficient PID
  Ready             True    KubeletReady            kubelet is posting ready status

A healthy node shows: MemoryPressure=False, DiskPressure=False, PIDPressure=False, Ready=True.

NotReady root causes:

Condition	Status=True	Root Cause	Fix
MemoryPressure	True	Node RAM >~85% full	Evict pods, add memory
DiskPressure	True	Node disk >~85% full	Clean images (crictl rmi), remove old logs
PIDPressure	True	Too many processes (>PID max)	Kill runaway processes
NetworkUnavailable	True	CNI plugin failed	Restart CNI, check node network config
Ready	False	kubelet stopped	systemctl restart kubelet

Kubelet Diagnostics

When a node is NotReady, the kubelet is either dead or reporting failure. Always check:

# On the node itself (via SSH or kubectl exec if accessible)
systemctl status kubelet
journalctl -u kubelet -n 50 --no-pager
journalctl -u kubelet --since "10 minutes ago" --no-pager

Common kubelet log errors:

# Certificate expired
E0605 10:00:00 kubelet.go:2448] Error getting node: Unauthorized
# → Rotate kubelet certificates

# Can't connect to API server
E0605 10:00:00 kubelet.go:2448] Unable to register node with API server
# → Check network connectivity to control plane, check kube-apiserver health

# Config error
Failed to load kubelet config file /var/lib/kubelet/config.yaml
# → Check kubelet config syntax

# CNI plugin failure
NetworkPlugin cni failed to set up pod network
# → Check CNI pod in kube-system is running

Node Maintenance — Cordon and Drain

kubectl cordon <node>      # Marks node Unschedulable
                           # ✓ Existing pods keep running
                           # ✗ New pods will not schedule here

kubectl drain <node>       # Cordons + evicts all evictable pods
  --ignore-daemonsets      # Required: DaemonSet pods can't be moved
  --delete-emptydir-data   # Required if any pod uses emptyDir volumes
  --grace-period=30        # Seconds to wait for graceful termination
  --timeout=120s           # Abort if drain takes too long

kubectl uncordon <node>    # Marks node Schedulable again

What drain does NOT evict:

DaemonSet-managed pods (use --ignore-daemonsets to skip them)
Pods with a PodDisruptionBudget that would be violated (drain will block and wait)
Mirror pods (static pods managed by kubelet directly)

Pod Eviction After NotReady

When a node goes NotReady, Kubernetes taints it with node.kubernetes.io/not-ready:NoExecute. Pods have a default tolerationSeconds: 300 for this taint — they tolerate the taint for 5 minutes before being evicted and rescheduled elsewhere.

←

Troubleshooting Networking

Troubleshooting the Control Plane

→