Troubleshooting the Control Plane

Control Plane Architecture

┌─────────────────────────── Control Plane Node ─────────────────────────────┐
│                                                                             │
│  /etc/kubernetes/manifests/            Static Pod Manifests                 │
│  ├── kube-apiserver.yaml    ─────────► API Server (port 6443)               │
│  ├── kube-scheduler.yaml    ─────────► Scheduler                            │
│  ├── kube-controller-manager.yaml ──► Controller Manager                   │
│  └── etcd.yaml              ─────────► etcd (port 2379)                     │
│                                                                             │
│  kubelet watches /etc/kubernetes/manifests/ and ensures these pods run.     │
│  They appear in: kubectl get pods -n kube-system                            │
│                                                                             │
│  Certificates:    /etc/kubernetes/pki/                                      │
│  kubeconfig:      /etc/kubernetes/admin.conf → ~/.kube/config               │
└─────────────────────────────────────────────────────────────────────────────┘

Diagnostic Sequence When kubectl Fails

kubectl get nodes returns: "The connection to the server 127.0.0.1:6443 was refused"
       │
       ├── Step 1: Is kube-apiserver running?
       │   kubectl get pods -n kube-system | grep apiserver
       │   crictl ps | grep apiserver            (on the control plane node)
       │
       ├── Step 2: Check the static pod manifest
       │   cat /etc/kubernetes/manifests/kube-apiserver.yaml
       │   Look for: typos in flags, wrong file paths, wrong etcd endpoint
       │
       ├── Step 3: Check kubelet logs for manifest errors
       │   journalctl -u kubelet -n 100 | grep -i "error|fail|apiserver"
       │
       ├── Step 4: Check kubeconfig
       │   cat ~/.kube/config | grep server
       │   Should be: https://127.0.0.1:6443 (or the control plane IP)
       │
       └── Step 5: Check etcd health
           ETCDCTL_API=3 etcdctl endpoint health \
             --endpoints=https://127.0.0.1:2379 \
             --cacert=/etc/kubernetes/pki/etcd/ca.crt \
             --cert=/etc/kubernetes/pki/etcd/server.crt \
             --key=/etc/kubernetes/pki/etcd/server.key

Static Pod Manifests — How They Work

The kubelet has a --pod-manifest-path (or staticPodPath in config.yaml) pointing to /etc/kubernetes/manifests/. The kubelet watches this directory:

• Add a file → kubelet creates the pod within seconds
• Edit a file → kubelet detects the change and recreates the pod
• Delete a file → kubelet terminates the pod

This is why control plane components survive API server restarts — they are managed directly by kubelet, not by the API server.

Important: Changes take effect when kubelet detects the file change. You do NOT need to restart kubelet for manifest changes.

Common Control Plane Failures

# kube-apiserver won't start after manifest edit
journalctl -u kubelet | grep kube-apiserver
# Error: open /etc/kubernetes/pki/apiserver.crt: no such file or directory
# → Wrong certificate path in manifest

# kube-apiserver: flag error
# flag provided but not defined: --etcd-serverss (typo)
# → Edit manifest, fix the typo, kubelet recreates the pod

# kube-scheduler: can't connect to API server
# Failed to get delegated authentication kubeconfig
# → kube-apiserver may be down, fix API server first

# etcd: member list fails
# context deadline exceeded
# → etcd data dir corrupted or network issue

Checking Control Plane Health

# Quick overview of control plane pods
kubectl get pods -n kube-system

# Component status (may show unhealthy for deprecated endpoints)
kubectl get componentstatuses

# Check individual component pod logs
kubectl logs -n kube-system kube-scheduler-<node-name>
kubectl logs -n kube-system kube-controller-manager-<node-name>

# etcd health check (from control plane node)
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# etcd member list
ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key