DaemonSets
Run exactly one Pod on every node for cluster-wide infrastructure like log collectors and monitoring agents.
NoSchedule taints to run system-critical pods (like network plugins or log agents) on master or cordoned nodes, and why does the default scheduler handle DaemonSet pods differently than normal application pods?What Is a DaemonSet?
A DaemonSet ensures that exactly one copy of a Pod runs on every node in the cluster (or a subset of nodes, if you use a nodeSelector or tolerations). It is the answer to the question: *"How do I run something on every machine in my cluster?"*
Automatic Scheduling
When you add a new node to the cluster, the DaemonSet controller automatically schedules the Pod on it — no manual action needed. When a node is removed or drained, the DaemonSet Pod on that node is garbage collected.
DaemonSet vs Deployment
| DaemonSet | Deployment | |
|---|---|---|
| Scheduling | 1 Pod per node | N replicas, anywhere |
| Scaling | Automatic (matches node count) | Manual or HPA |
| Use case | Node-level infrastructure | Application workloads |
Real-World Use Cases
- Log collectors: Fluentd, Filebeat — collect logs from every node's /var/log
- Monitoring agents: Prometheus node-exporter — scrape hardware metrics from each node
- Network plugins: Calico, Cilium — must run on every node to manage the overlay network
- Storage daemons: Ceph, GlusterFS agents
- Security agents: Falco, intrusion detection
Update Strategies
| Strategy | Behaviour |
|---|---|
RollingUpdate (default) | Updates one node at a time; the old Pod is deleted before the new one starts |
OnDelete | New Pod version is only applied when you manually delete the old Pod — gives full control over timing |
RollingUpdate is safe for most workloads. Use OnDelete when you need to coordinate the rollout with node maintenance windows.
DaemonSet Controller and Scheduling Engine
+-------------------------------------------------------------+
| DaemonSet Controller |
+------------------------------+------------------------------+
| Watches Nodes & Pods
v
+-----------------------------------------------+
| Injects NodeAffinity & Critical Tolerations |
+------------------------+----------------------+
|
v
+-------------------------------------------------------------+
| kube-scheduler |
| (Bypasses cordons/taints using injected tolerations) |
+----+-------------------------+-------------------------+----+
| | |
v Node A v Node B v Master Node (Tainted)
+----+------------+ +----+------------+ +----+------------+
| log-collector | | log-collector | | log-collector |
| Pod (Ready) | | Pod (Ready) | | Pod (Ready) |
+-----------------+ +-----------------+ +-----------------+Under the Hood: DaemonSet Scheduling & Node Taints
1. Modern Scheduler Integration
Historically, the DaemonSet controller set the Pod's spec.nodeName directly. In modern versions (since v1.12), the controller delegates scheduling to the default kube-scheduler. The controller automatically appends a NodeAffinity corresponding to the target node's labels, ensuring the scheduler processes the Pod correctly.
2. Bypassing Master and Cordoned Node Taints
To make sure system-level daemons run everywhere, the DaemonSet controller automatically injects critical tolerations to bypass taints:
node.kubernetes.io/unschedulable(allows running on cordoned/drained nodes)node.kubernetes.io/not-ready/node.kubernetes.io/unreachable(allows scheduling during transient network/host outages)node.kubernetes.io/memory-pressure/node.kubernetes.io/disk-pressure/node.kubernetes.io/pid-pressure(allows scheduling during resource crunch)
3. Resource Allocation and Quality of Service (QoS)
System daemons should always be defined with Guaranteed QoS (requests = limits) and given system PriorityClasses (system-node-critical or system-cluster-critical). Kubelet adjusts their oom_score_adj value (typically to -997), preventing them from being killed by the OOM Killer or evicted during node-level resource starvation.