Node Management — Cordon, Drain & Maintenance

Safely remove nodes from rotation for maintenance without disrupting running workloads.

intermediate60 minRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: You need to reboot a node for a kernel upgrade. 5 pods are running on it. How do you ensure the pods are safely rescheduled before you reboot, without any downtime?

The Maintenance Workflow

1. kubectl cordon <node>   → mark Unschedulable (no NEW pods)
2. kubectl drain <node>    → evict all pods + cordon
3. [perform maintenance]
4. kubectl uncordon <node> → re-enable scheduling

Cordon vs Drain

Command	Effect
`cordon`	Marks node Unschedulable. Existing pods keep running
`drain`	Evicts all pods AND marks node Unschedulable
`uncordon`	Marks node Schedulable again

kubectl drain Flags

kubectl drain node-1 \
  --ignore-daemonsets \   # DaemonSet pods cannot be moved — skip them
  --delete-emptydir-data  # Allow draining pods with emptyDir volumes

Without --ignore-daemonsets, drain fails if DaemonSet pods are present.

Taints and Tolerations

Taints repel pods from nodes. Tolerations allow pods to be scheduled on tainted nodes.

Node taint: key=value:effect
  - NoSchedule: new pods without toleration not scheduled
  - PreferNoSchedule: soft preference against scheduling
  - NoExecute: existing pods without toleration are evicted

Pod toleration:
  key: "key"
  operator: "Equal"
  value: "value"
  effect: "NoSchedule"

Node Conditions

kubectl describe node shows:

Ready — node is healthy
MemoryPressure — node is low on memory
DiskPressure — node is low on disk
PIDPressure — too many processes
NetworkUnavailable — CNI not configured

←

SecurityContext — Pod & Container Hardening

Admission Controllers — Request Validation & Mutation

→