Health Probes

Why Probes Matter

Without health probes, the Kubernetes control plane relies solely on the container process exit code. If a process enters an infinite loop, deadlocks, or runs out of database connections but does not crash, the PID remains alive. To the container runtime, the container is healthy. Probes allow the kubelet to query application-level health directly.

Visualizing Probes Lifecycle & Traffic Routing

               +------------------------------------------------+
               |                  Kubelet                       |
               +------------------------------------------------+
                     | (probes container at periodSeconds)
                     |
         +-----------+-----------+
         | (httpGet / tcpSocket) | (exec)
         v                       v
+------------------+   +---------------------------------------+
|  Host network    |   | CRI gRPC API (ExecSync)               |
|  namespace check |   | -> calls container runtime (containerd)|
+------------------+   | -> runs process via runc in container |
         |             +---------------------------------------+
         |                                |
         +---------------++---------------+
                         |
                         v
                Does probe pass?
                 /                           NO              YES
               /                   +--------------------+     +--------------------------------+
   |  Which probe?      |     | Pod marked Ready / Healthy     |
   |   /               |     +--------------------------------+
  Liveness       Readiness
    /                Restart container  Remove Pod IP from EndpointSlice
                   -> kube-proxy removes iptables/IPVS rule

How Kubelet Executes Probes at the OS Level

Kubelet runs a dedicated Prober Manager that schedules checks per container. The execution depends on the mechanism:

• httpGet / tcpSocket / gRPC: The kubelet initiates the request from the host network namespace directly to the container's IP address. For httpGet, any response code >= 200 and < 400 is a success.
• exec: Kubelet makes a gRPC request to the Container Runtime Interface (CRI) API (ExecSync). The container runtime (e.g., containerd) invokes the OCI runtime (e.g., runc) to execute the binary within the container’s PID, mount, and network namespaces. The probe succeeds if the process returns exit code 0. This is resource-intensive, as it forks a process inside the container every few seconds.

Probe Types and Internal Pipelines

1. Startup Probe

• Purpose: Checks if the application has completed initialization.
• Workflow: While the startup probe is running, all other probes (liveness, readiness) are disabled. If the startup probe fails failureThreshold times, the kubelet kills the container and initiates the restart policy.

2. Liveness Probe

• Purpose: Detects if the application has entered a non-recoverable deadlocked state.
• Workflow: If the probe fails, kubelet communicates with the CRI to terminate the container. The termination process triggers a SIGTERM, waits for terminationGracePeriodSeconds, and sends SIGKILL. A new container is started.

3. Readiness Probe

• Purpose: Determines if the container is ready to accept client traffic.
• Workflow: Unlike liveness, readiness failure does not trigger a restart. Instead:

1. The kubelet updates the Pod's status to Ready: False.

2. The EndpointSlice controller in the control plane detects this state change and removes the Pod's IP address from the corresponding EndpointSlice objects.

3. kube-proxy on all nodes watches for EndpointSlice changes and recalculates the node's iptables or IPVS rules, omitting this Pod's IP. Traffic stops routing to the Pod.

Probe Mechanisms

• httpGet: sends an HTTP GET request — suitable for web services
• tcpSocket: checks if a TCP port accepts connections — suitable for databases, message queues
• exec: runs a command inside the container — most flexible, any exit 0 = success
• grpc: checks a gRPC health endpoint (standard grpc.health.v1.Health) — added in Kubernetes 1.24

Key Parameters

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10   # wait before first probe
  periodSeconds: 10          # probe every N seconds
  failureThreshold: 3        # fail N times before action
  successThreshold: 1        # pass N times to be considered healthy
  timeoutSeconds: 1          # how long to wait for a probe response before counting it as a failure

Classic Mistake

Missing readiness probe during a rolling update: Kubernetes removes old pods before new pods have actually warmed up, causing a brief window of 502 errors. A readiness probe prevents this — new pods only receive traffic after the probe passes.