Custom Resource Definitions (CRDs)

The Kubernetes API is Extensible

The built-in Kubernetes resource types (Pod, Deployment, Service, ConfigMap…) are not hardcoded into the API server. They are registered resource definitions. And you can add your own.

Custom Resource Definitions (CRDs) let you register new resource kinds with the API server. Once registered, your custom resources are full citizens of the Kubernetes API — you can kubectl get, kubectl apply, kubectl delete, and kubectl describe them just like Pods.

CRD Controller Reconciliation Architecture

+-------------------------------------------------------------+
|                     Kubernetes API Server                   |
|  +----------------+      +--------+      +---------------+  |
|  | Mutating/Valid | ---> |  etcd  | ---> | HTTP/2 Watch  |  |
|  | Webhooks       |      +--------+      | stream        |  |
+--+----------------+----------------------+-------+-------+--+
                                                   |
                                                   | (Informer / Reflector)
                                                   v
+--------------------------------------------------+----------+
|                 Operator Controller Process                 |
|  +-------------------+        +--------------------------+  |
|  |   Local Cache     | <----> |   SharedIndexInformer    |  |
|  |   (Lister)        |        +------------+-------------+  |
|  +-------------------+                     |                |
|                                            v (events: add/upd/del)
|  +-------------------+        +--------------------------+  |
|  |  Reconcile Loop   | <----- | Rate-Limiting WorkQueue  |  |
|  | (Desired vs Actual|        +--------------------------+  |
|  +---------+---------+                                      |
+------------|------------------------------------------------+
             |
             | (1. Read Desired State / 2. Inspect Actual)
             v
+-------------------------------------------------------------+
|                         Real World                          |
|  +--------------------+        +-------------------------+  |
|  | Cloud Infrastructure|        | Cluster Resources       |  |
|  | (AWS RDS, LB, etc.) |        | (Pods, Services, etc.)  |  |
|  +--------------------+        +-------------------------+  |
+-------------------------------------------------------------+

CRD vs CR

The relationship: CRD is to CR as a Go struct is to a Go variable.

CRDs Alone Do Nothing

This is the crucial insight. When you create a CRD and then create a CR, the API server:

1. Validates the CR against the CRD schema
2. Stores the CR in etcd
3. Does nothing else

No actual database is created. No process is started. The CR is just structured data in etcd. To make something happen, you need a controller.

Low-Level Controller Architecture: Informers and Finalizers

Custom controllers (often written in Go using controller-runtime or client-go) run as workloads in the cluster and implement a specific reconciliation pattern:

1. SharedIndexInformer: Instead of querying the API server continuously, the controller uses an Informer. The Informer opens an HTTP/2 chunked transfer stream (Watch API) to the API server, caches the retrieved objects in a local thread-safe index (Lister), and propagates event hooks when objects are added, updated, or deleted.
2. WorkQueue: Events are pushed to a rate-limiting work queue. This queues tasks, merges duplicate keys, and retries reconciliations using exponential backoff when errors occur.
3. Finalizers (metadata.finalizers): To clean up external resources (like cloud databases or disks) before a CR is deleted from etcd, the controller adds a finalizer string to the CR. When a user deletes the CR, the API server sets a deletionTimestamp but blocks deletion until all finalizers are removed. The controller detects the deletion timestamp, cleans up the external infrastructure, and then removes its finalizer, allowing the CR to be purged from etcd.
4. Status Subresource (/status): CRDs expose a dedicated /status subresource. This isolates the status updates from spec changes. Spec edits increment metadata.generation, triggering a reconcile. Status updates do not. RBAC rules can restrict normal users from modifying the status, ensuring only the controller has permission to write status fields.

The Controller Pattern

A controller is a reconciliation loop:

loop forever:
  desired = read CR from API server
  actual  = observe current state (running processes, resources)
  if desired != actual:
    take action to move actual → desired

CRD Schema Validation

CRDs support OpenAPI v3 schema validation. The API server rejects CRs that violate the schema at admission time — before they are stored in etcd:

schema:
  openAPIV3Schema:
    type: object
    properties:
      spec:
        type: object
        required: ["engine", "version"]
        properties:
          engine:
            type: string
            enum: ["postgresql", "mysql", "redis"]
          version:
            type: string
          storage:
            type: string

Operator Pattern

A Kubernetes Operator is a custom controller that extends Kubernetes to manage complex, stateful applications. It packages operational knowledge (how to deploy, scale, backup, upgrade an application) into code that runs as a controller inside the cluster.

How Operators Work

Custom Resource (CR)
       │
       │  create/update/delete
       ▼
Operator Controller (watches CRs)
       │
       │  reconcile loop
       ▼
Kubernetes API (creates Deployments, Services, Secrets, etc.)

The operator watches for Custom Resources of a specific Kind and reconciles the cluster state to match the CR's spec — the same control loop pattern used by the built-in Deployment controller.

Operator Frameworks

Installing Existing Operators

You usually install operators via:

1. Helm chart (most common):

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

1. Operator Lifecycle Manager (OLM) — a system for managing operator installation and upgrades from OperatorHub:

kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.28.0/crds.yaml
kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.28.0/olm.yaml

1. OperatorHub.io — the catalog of community operators (similar to Docker Hub but for operators)

Why Operators vs Plain Helm?