etcd — Backup, Restore & Health

Understand etcd as the cluster's single source of truth, then master the snapshot backup and restore procedure that appears on every CKA exam.

advanced5 hoursRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: etcd uses the Raft consensus algorithm. If you have a 3-node etcd cluster and 2 nodes fail, what happens to write operations? What about read operations with stale reads allowed? Think about the difference between availability and consistency before reading.

etcd: The Cluster Brain

etcd is a distributed, strongly consistent key-value store. In Kubernetes, it is the only persistent storage layer — every object you create with kubectl (pods, deployments, secrets, configmaps) is ultimately a serialized protobuf blob stored in etcd. If etcd is lost without a backup, the cluster state is gone permanently.

Raft Consensus and Quorum

etcd uses the Raft algorithm to keep all cluster members consistent:

3-node etcd cluster (quorum = 2):

  Leader ◄──── heartbeat ────► Follower-1
     │                              │
     └──────── heartbeat ──────► Follower-2

  Write request → Leader proposes → 2/3 nodes acknowledge → committed
  1 node can fail and the cluster remains available

5-node etcd cluster (quorum = 3):
  Can tolerate 2 simultaneous failures

etcd Ports

Port	Purpose
2379	Client traffic (kube-apiserver → etcd, etcdctl → etcd)
2380	Peer-to-peer Raft replication (etcd nodes talk to each other)

ETCDCTL_API=3: Why It Matters

etcdctl supports two API versions. The v2 API is deprecated and behaves differently:

# Wrong (v2 API — different command syntax, incompatible snapshots):
etcdctl backup ...

# Correct (v3 API — required for all Kubernetes operations):
export ETCDCTL_API=3
etcdctl snapshot save ...

Always export ETCDCTL_API=3 at the start of any CKA task involving etcd.

Certificate Flags (Memorize These)

All etcdctl v3 commands that contact the live etcd cluster require four TLS flags:

--endpoints=https://127.0.0.1:2379          # etcd client port on control plane
--cacert=/etc/kubernetes/pki/etcd/ca.crt    # cluster CA for etcd
--cert=/etc/kubernetes/pki/etcd/server.crt  # client certificate
--key=/etc/kubernetes/pki/etcd/server.key   # client private key

Backup and Restore Flow

BACKUP (no downtime required):
  etcdctl snapshot save /backup/etcd-snapshot.db
      │
      └─▶ etcd streams a consistent point-in-time snapshot to the file
          Cluster continues running normally during backup

RESTORE (requires control plane downtime):

  Step 1: Move static pod manifests out (stops apiserver, scheduler, controller-manager, etcd)
    mv /etc/kubernetes/manifests/*.yaml /tmp/k8s-manifests/

  Step 2: Restore snapshot to a new data directory
    etcdctl snapshot restore /backup/etcd-snapshot.db --data-dir=/var/lib/etcd-restore

  Step 3: Update etcd static pod manifest to use new data directory
    Edit /tmp/k8s-manifests/etcd.yaml: change --data-dir value

  Step 4: Move manifests back (kubelet restarts control plane)
    mv /tmp/k8s-manifests/*.yaml /etc/kubernetes/manifests/

  Step 5: Wait for API server to come back and verify
    kubectl get all

Why Stop the API Server During Restore?

If the API server is running while you restore etcd, it will continue writing new state to the old etcd instance. When you swap to the restored snapshot, those in-flight writes are lost and you have a split-brain scenario. Always halt the control plane first.

←

Bootstrapping a Cluster with kubeadm

Upgrading a Cluster with kubeadm

→