Production Readiness Checklist

Production is Different

A cluster that works in development often fails in production in surprising ways. The difference is not just traffic volume — it is the accumulation of small omissions: no resource limits, no probes, no network policies, no log aggregation.

This module ties together every topic from the course into a production readiness checklist.

Pod Quality of Service (QoS) & Eviction Hierarchy

                EVICTION / OOM-KILL PRIORITY HIERARCHY

     +----------------------------------------------------+  - High Priority Eviction
     | BestEffort QoS                                     |  - oom_score_adj: 1000
     | (No resource requests or limits configured)        |  - First to be terminated
     +----------------------------------------------------+
     | Burstable QoS                                      |  - Medium Priority Eviction
     | (Requests set; limits optionally set but not equal)|  - oom_score_adj: dynamic
     |                                                    |    (1000 - 10 * req/cap)
     +----------------------------------------------------+
     | Guaranteed QoS                                     |  - Lowest Priority Eviction
     | (Requests == Limits for all CPUs and Memory)       |  - oom_score_adj: -997
     |                                                    |  - Shielded from OOM-kill
     +----------------------------------------------------+

Low-Level Eviction Mechanics & Linux OOM Configuration

When a node runs low on physical memory or disk space, the kubelet Eviction Manager initiates hard or soft evictions to keep the node OS stable:

1. Eviction Thresholds: By default, kubelet monitors resource usage and triggers evictions when memory availability falls below 100Mi or node disk storage is less than 10%.
2. QoS Classes & Eviction Ranking: The kubelet ranks pods for eviction according to their resource usage relative to their requests, sorted by QoS class:

- BestEffort: Pods with no CPU/memory requests or limits set. The kubelet configures their containers with Linux /proc//oom_score_adj set to 1000. These are the first to be terminated.

- Burstable: Pods with requests set, but limits do not match requests or are omitted. The oom_score_adj is computed dynamically as 1000 - (10 * memory_request / node_capacity). If they consume more than their requested memory, they are evicted before Guaranteed pods.

- Guaranteed: Pods where requests equal limits for all CPUs and memory across all containers. The oom_score_adj is set to -997. These are protected and only evicted if the node runs completely out of memory and no other lower-priority pods exist.

1. PodDisruptionBudget (PDB) Limits: PDBs (e.g. maxUnavailable: 1) are respected during voluntary cluster admin actions (like kubectl drain) by using the API server's Eviction endpoint. However, during node-pressure crises, the local kubelet executes involuntary evictions directly and ignores PDBs to protect node integrity.
2. Topology Spread Constraints: In production, workloads should leverage topology spread constraints and anti-affinity rules to distribute pods across hostnames or availability zones, ensuring single-node or single-zone failures don't cause complete service outage.
3. API Token Hardening: Disabling ServiceAccount token mounting (automountServiceAccountToken: false) ensures that compromised containers do not have immediate access to the API server credentials at /var/run/secrets/kubernetes.io/serviceaccount/token.

Checklist: Workload Configuration

Every production Pod must have:

• ✅ Resource requests and limits on every container — without requests, the scheduler cannot place pods correctly; without limits, one runaway pod can OOMKill its neighbors
• ✅ Liveness probe — tells Kubernetes when to restart a container that is deadlocked or hung
• ✅ Readiness probe — tells Kubernetes when the container is ready to receive traffic (prevents requests to a pod still starting up)
• ✅ Multiple replicas for all stateless apps — a single-replica deployment has zero tolerance for node failure or rolling update disruption
• ✅ PodDisruptionBudget for critical services — limits voluntary disruptions (node drains, cluster upgrades) to maintain availability
• ✅ Pod anti-affinity — spread replicas across nodes (or availability zones) so a single node failure does not take all replicas offline

Checklist: Networking & Security

• ✅ Services use ClusterIP internally; Ingress for external HTTP/HTTPS with TLS termination
• ✅ NetworkPolicies: default-deny all ingress/egress, then explicit allow rules for required traffic
• ✅ Secrets for all sensitive data — never hard-code passwords in ConfigMaps or environment variables
• ✅ RBAC: dedicated ServiceAccounts per application, least-privilege Roles — no wildcard verbs
• ✅ automountServiceAccountToken: false on every Pod that does not call the Kubernetes API

Checklist: Reliability

• ✅ Namespaces + ResourceQuotas per team or environment — prevent one team from starving others
• ✅ Pod topology spread constraints for zone-aware scheduling
• ✅ HPA for traffic-sensitive services — scale out automatically under load
• ✅ Cluster Autoscaler for node-level scaling — let the cluster grow and shrink with demand

Checklist: Observability

• ✅ metrics-server + Prometheus + Grafana — real-time and historical metrics
• ✅ Centralized log aggregation (Loki/ELK) — logs survive pod deletion
• ✅ Alerts configured for: pod CrashLoopBackOff, OOMKilled, high error rates, disk pressure, certificate expiry

Checklist: Operations

• ✅ Regular etcd backups — etcd is the source of truth; losing it means losing the entire cluster state
• ✅ Cluster upgrade plan — Kubernetes supports N-2 version skew; upgrade one minor version at a time, control plane before nodes
• ✅ Helm or Kustomize for all deployments — no manual kubectl apply in production
• ✅ GitOps (ArgoCD or Flux) — Git is the source of truth; every cluster change goes through pull request review and is automatically reconciled

GitOps

GitOps means: the desired state of the cluster is declared in Git, and an agent (ArgoCD, Flux) continuously reconciles the cluster toward that state.

Benefits:

• Auditability: every change has a Git commit, author, and timestamp
• Rollback: git revert undoes any change
• Drift detection: the agent alerts when cluster state diverges from Git
• No direct kubectl access needed in production — all changes go through Git