StatefulSets

Run databases and clustered apps that need stable identities, ordered startup, and per-Pod persistent storage.

intermediate75 minRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: When a StatefulSet Pod (e.g., mysql-1) crashes or its host node becomes unreachable, how does Kubernetes guarantee that a replacement Pod doesn't start on another node and access the same raw block storage volume concurrently, potentially causing data corruption?

Why Not Deployments for Stateful Apps?

Deployments are designed for stateless workloads. Every Pod in a Deployment is interchangeable — they get random names (web-7d4f9b-xk2p) and random IPs. If a Pod dies, a replacement with a completely different name and IP is created.

Databases are different. A MySQL replica needs to know exactly which other replica is the primary. A Kafka broker needs a stable identity so other brokers can find it by name. Random Pod names and IPs break clustering protocols.

StatefulSet Guarantees

A StatefulSet provides four guarantees Deployments cannot:

1. Stable Pod Names

Pods are numbered from zero: mysql-0, mysql-1, mysql-2. These names survive restarts — if mysql-1 dies, its replacement is also called mysql-1.

2. Stable Network Identity via Headless Service

A Headless Service (clusterIP: None) does not get a virtual IP. Instead, DNS returns the actual Pod IPs directly. Combined with a StatefulSet, each Pod gets its own stable DNS entry:

mysql-0.mysql.default.svc.cluster.local
mysql-1.mysql.default.svc.cluster.local
mysql-2.mysql.default.svc.cluster.local

These DNS names are stable — other services can always reach a specific replica.

3. Ordered Startup and Shutdown

Pods start in order: 0 → 1 → 2. Each Pod must be Running and Ready before the next one starts. Shutdown happens in reverse: 2 → 1 → 0. This lets databases elect a primary before replicas start.

4. Per-Pod PVCs via volumeClaimTemplates

Instead of sharing one PVC, volumeClaimTemplates creates a separate PVC per Pod:

data-web-0 → mounted by web-0
data-web-1 → mounted by web-1
data-web-2 → mounted by web-2

If web-1 is deleted and recreated, it rebinds to data-web-1 — its own dedicated data.

When to Use StatefulSets

Use StatefulSets for: MySQL, PostgreSQL, Cassandra, Redis Cluster, Kafka, ZooKeeper, Elasticsearch.

Use Deployments for: stateless apps — web servers, APIs, workers. StatefulSets add complexity you do not need for stateless workloads.

StatefulSet Cluster Architecture & DNS Resolution

        +-------------------------------------------------+
        |              Headless Service                   |
        |              (clusterIP: None)                  |
        +-----------------------+-------------------------+
                                |
          +---------------------+---------------------+
          | (DNS Lookup: web.default.svc.cluster.local) |
          v                                           v
  +------------------+                       +------------------+
  |  Pod: web-0      |                       |  Pod: web-1      |
  |  IP: 10.244.1.5  |                       |  IP: 10.244.2.9  |
  +--------+---------+                       +--------+---------+
           |                                          |
  Matches  | (Deterministic                           | (Deterministic
  Ordinal  |  Binding)                       Matches  |  Binding)
           v                                 Ordinal  v
  +--------+---------+                       +--------+---------+
  | PVC: data-web-0  |                       | PVC: data-web-1  |
  | (Bound to PV-0)  |                       | (Bound to PV-1)  |
  +------------------+                       +------------------+

Under the Hood: StatefulSet Controller & Safety Guarantees

1. The Controller Sync Loop

The StatefulSet controller operates using a deterministic state sync loop. It uses the Ordinal Index to map Pods to PVCs. The stable hostname format is:

...svc.cluster.local

CoreDNS dynamically manages these hostnames via Endpoint/EndpointSlice controllers, enabling direct member-to-member clustering communication.

2. The "At-Most-One-Pod" Safety Guarantee & Partitions

If a node hosting a StatefulSet Pod (e.g., web-1) loses connection to the API server, it enters Unknown state.

Unlike a Deployment, the StatefulSet controller will never automatically force-delete or reschedule the Pod onto a new node.
Since the underlying physical volume (e.g., cloud block storage like AWS EBS) is mapped to ReadWriteOnce, attaching the volume to a replacement Pod on another node while the partitioned node might still be running and writing to it could cause severe data corruption.
The controller waits for the node to return or for manual administrative intervention via kubectl delete pod web-1 --force --grace-period=0.

←

Volumes & PersistentVolumes

DaemonSets

→