Taints, Tolerations & Node Affinity

The Kubernetes Scheduling Pipeline

The kube-scheduler runs as a control plane loop, matching newly created, unscheduled Pods (spec.nodeName is blank) to the most appropriate node in the cluster. It achieves this in three consecutive phases:

1. Filtering (Predicates): Filters out nodes that do not satisfy the Pod's hardware requirements or constraints.
2. Scoring (Priorities): Ranks the remaining nodes by scoring them (from 0 to 100) using priority functions (e.g., node affinity weight, image locality, balanced resource usage).
3. Binding: The scheduler selects the highest-scoring node and writes a binding object to the API server, which sets the Pod's spec.nodeName. The node's local kubelet then detects this assignment and spins up the container via the CRI gRPC API.

The Scheduling Pipeline

Incoming Pod Spec (Constraints)

│

▼

┌───────────────────────────────────────────────────────────┐

│ SCHEDULER FILTERING PHASE │

│ - Node Taints vs Pod Tolerations │

│ - NodeSelectors & Node Affinity (requiredDuring...) │

│ - Resource limits/requests vs Node Allocatable capacity │

│ - Pod Topology Spread & Co-location constraints │

└─────────────────────────────┬─────────────────────────────┘

│

Filtered Nodes (Survivors)

│

▼

┌───────────────────────────────────────────────────────────┐

│ SCHEDULER SCORING PHASE │

│ - Node Affinity Weighting (preferredDuring...) │

│ - Image Locality (prefer nodes with image already cached) │

│ - Inter-pod Affinity / Anti-affinity scoring │

│ - Balanced Resource Allocation (CPU/Memory spread) │

└─────────────────────────────┬─────────────────────────────┘

│

Highest-Scored Node Chosen

│

▼

Bind Request

(Writes spec.nodeName via API)

Node Taints and Pod Tolerations

Taints are applied to nodes to allow them to repel a set of Pods. Tolerations are applied to Pods to allow (but not force) them to schedule on nodes with matching taints.

Format: kubectl taint nodes key=value:effect

Eviction Delay via tolerationSeconds

If a Pod tolerates a NoExecute taint (e.g., taints applied by the Node Lifecycle Controller like node.kubernetes.io/unreachable during network partitions), it can specify a tolerationSeconds limit:

tolerations:
- key: "node.kubernetes.io/unreachable"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 300

This tells Kubernetes: "If this node becomes unreachable, let my Pod stay running on it for 5 minutes before evicting it."

Expressive Node Selection

nodeSelector

The simplest constraint. Requires an exact match of key-value labels on the node:

spec:
  nodeSelector:
    disktype: ssd

Node Affinity

The advanced successor to nodeSelector. Supports logical operators (In, NotIn, Exists, DoesNotExist, Gt, Lt) and contains two categories:

• Hard: requiredDuringSchedulingIgnoredDuringExecution (Must match)
• Soft: preferredDuringSchedulingIgnoredDuringExecution (Weighted priority from 1 to 100, scheduler scores these nodes higher)

The *IgnoredDuringExecution* suffix means that if node labels change after scheduling, the Pod remains running (does not get evicted).

Inter-Pod Affinity & Anti-Affinity

These rules constraint scheduling based on labels of Pods already running on nodes in specific topology domains.

• Topology Domains: Defined by topologyKey (e.g., kubernetes.io/hostname for nodes, or topology.kubernetes.io/zone for cloud availability zones).
• Anti-Affinity: Crucial for high-availability. Prevents co-locating replicas on the same hardware or zone.