PodDisruptionBudgets & Cluster Maintenance

Voluntary vs Involuntary Disruptions

Workload availability is subject to two classes of disruptions:

PodDisruptionBudgets (PDBs) are validation policies that *only* protect against voluntary disruptions. They have no control over network partitions or failed hardware.

PodDisruptionBudget (PDB) Mechanics

A PDB defines a minimum availability threshold that the API server enforces during voluntary disruptions. When a controller or administrator attempts to drain a node, it does not delete Pods directly. Instead, it sends an HTTP POST request to the Pod's eviction subresource endpoint (e.g., /api/v1/namespaces/default/pods/my-pod/eviction).

The API server intercepts this eviction call, checks all active PDBs, and:

• Allows the eviction if the number of running, Ready pods (those satisfying minReadySeconds) remains above the PDB threshold.
• Rejects the eviction (returning an HTTP 429 Too Many Requests error) if it would violate the budget. The drain utility then sleeps and retries the request.

Eviction Lifecycle & PDB Enforcement during Node Drain

[ Admin / Script ] ──────────────► [ kubectl drain node-1 ]

│

▼ (Sends Cordon Request)

[ node.spec.unschedulable = true ]

│

▼ (Loops over non-DaemonSet Pods)

[ POST /eviction API ]

│

▼

┌────────────────────────┐

│ API SERVER │

│ - Evaluates PDBs │

└──────────┬─────────────┘

│

┌──────────────┴──────────────┐

Allowed? Yes│ │No (Violates Budget)

▼ ▼

[ Deletes Pod Object ] [ Returns 429 Conflict ]

│ │

▼ ▼

[ Kubelet ] [ kubectl drain ]

(Sends SIGTERM, waits grace period, (Sleeps 5s, retries

runs container preStop lifecycle) eviction request)

Configuring PDB Specs

You specify either minAvailable or maxUnavailable (never both):

• minAvailable: An absolute number (e.g., 2) or percentage (e.g., 80%) of Pods that must remain healthy.
• maxUnavailable: An absolute number or percentage of Pods that can be concurrently terminated.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2 # At least 2 pods must remain Ready
  selector:
    matchLabels:
      app: web

Node Maintenance Lifecycle (Cordon & Drain)

Upgrading and maintaining nodes follows a strict three-phase lifecycle:

1. Cordon: Marks the node as unschedulable (node.spec.unschedulable = true). Running Pods are unaffected, but no new Pods can be scheduled on this node.
2. Drain: Evicts all non-DaemonSet Pods from the node using the eviction API.

- DaemonSets: Drained pods are ignored by default since they run per-node utilities and are managed by the DaemonSet controller. You must pass --ignore-daemonsets to let the drain proceed.

- Local Data: Pods using emptyDir volumes require the --delete-emptydir-data flag since their local storage will be discarded.

1. Uncordon: After physical maintenance or OS upgrades, uncordon the node to make it schedulable again.

Graceful Termination Internals

Once an eviction is approved, the API server deletes the Pod object. The node's kubelet detects the deletion event and coordinates graceful termination:

1. PreStop Hook: The kubelet executes any configured preStop lifecycle hook inside the container. This is synchronous.
2. SIGTERM: The container runtime sends the SIGTERM signal (PID 1) to the container process, giving it time to close network sockets, complete requests, or persist buffers.
3. Grace Period: The kubelet waits for the terminationGracePeriodSeconds (default: 30s).
4. SIGKILL: If the process is still running after the grace period, the runtime issues SIGKILL to forcibly terminate it.