4/55 hours
Cluster Ops
PKI & Certificate Management
Understand the Kubernetes PKI hierarchy, inspect certificate expiry dates, renew certificates before they expire, and trace how kubeconfig files embed certificates for component authentication.
advanced5 hoursRead → Lab → Quiz → Practice
🧠 Brain Warm-Up
🧠 Brain Warm-Up: A kubeadm cluster was bootstrapped one year ago and no certificate maintenance has been performed. The cluster suddenly becomes unreachable — kubectl commands return "tls: certificate has expired or is not yet valid". Which certificates likely expired, and what is the fastest path to recovery? Think before reading.
Kubernetes PKI Architecture
When kubeadm init runs, it generates a complete Public Key Infrastructure with three separate root Certificate Authorities:
/etc/kubernetes/pki/
│
├─ ca.crt / ca.key ← Cluster root CA
│ Signs: apiserver.crt, apiserver-kubelet-client.crt,
│ all kubeconfig client certs (admin, controller-manager, scheduler)
│
├─ etcd/
│ ├─ ca.crt / ca.key ← etcd root CA (separate from cluster CA)
│ │ Signs: etcd server cert, etcd peer certs, apiserver-etcd-client.crt
│ ├─ server.crt / server.key
│ ├─ peer.crt / peer.key
│ └─ healthcheck-client.crt
│
├─ front-proxy-ca.crt / front-proxy-ca.key ← Front proxy CA
│ Signs: front-proxy-client.crt (used by aggregation layer)
│
└─ sa.key / sa.pub ← ServiceAccount signing keypair
(NOT a certificate — used to sign/verify JWT tokens)Certificate Lifetime
By default, kubeadm issues certificates with a 1-year validity period. The cluster CA itself has a 10-year validity. This means:
- Certificates must be renewed before they expire (ideally 30–60 days in advance)
- kubeadm certs check-expiration shows days until expiry for every certificate
- kubeadm certs renew all renews everything in one command, then each component must be restarted
Certificate Authentication vs ServiceAccount Tokens
| Aspect | X.509 Certificate | ServiceAccount Token (JWT) |
|---|---|---|
| Who uses it | Control plane components, human admins | Pods, in-cluster apps |
| Storage | /etc/kubernetes/pki/ and kubeconfigs | Projected into pods as files |
| Expiry | 1 year (kubeadm default) | Configurable, auto-rotated by kubelet |
| Validation | TLS handshake, cert chain verification | OIDC JWT signature verification by apiserver |
| Rotation | Manual (kubeadm certs renew) | Automatic (kubernetes.io/service-account-token) |
kubeconfig Files and Certificate Embedding
Each kubeconfig file embeds:
- The cluster CA cert (to verify the API server's TLS cert)
- A client certificate and key (base64 encoded)
- The API server URL
kubectl config view --raw
│
└─ certificate-authority-data: <base64 of ca.crt>
client-certificate-data: <base64 of apiserver-kubelet-client.crt>
client-key-data: <base64 of key>Recovery from Expired Certificates
If certificates expire:
kubeadm certs renew all— regenerates all certificates- Restart each static pod (move manifests out and back in, or:
crictl rm --all) - Regenerate kubeconfigs if needed:
kubeadm init phase kubeconfig all - Copy the new admin.conf to ~/.kube/config