Volumes & PersistentVolumes

Ephemeral vs Persistent Storage: The Filesystem Boundary

A container's root filesystem is ephemeral, managed via an overlay filesystem (e.g., overlay2) where writes are recorded in a temporary container-specific write layer. When the container process exits and is re-created, this write layer is discarded. To preserve state, Kubernetes uses decoupled storage primitives.

Basic Volume Types

#### emptyDir

An ephemeral scratch directory provisioned directly on the host node's storage media (typically backed by the node's primary disk under /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/ or memory if medium: Memory is specified to create a tmpfs RAM disk).

• Scope: Tied directly to the Pod's lifecycle. It survives container crashes and restarts, but is destroyed when the Pod is terminated or rescheduled.
• Use Case: Shared memory/disk workspace for multi-container pods (e.g., a sidecar pattern where an app writes logs and a Fluent Bit container reads them).

volumes:
  - name: scratch
    emptyDir: {}
containers:
  - volumeMounts:
    - name: scratch
      mountPath: /scratch

#### hostPath

Mounts a specific directory or file from the host node's filesystem directly into the container's mount namespace.

• Scope: Node-specific. It is not portable; if the Pod is rescheduled to another node, it has no access to the previous node's data.
• Security Risks: Bypasses container namespace boundaries. If a container runs as root or has write permissions, a compromised Pod can read/write raw system configurations, docker socket files, or SSH keys on the host. It should be disabled via PodSecurityStandards/Admission controllers in production.

Persistent Storage: PV, PVC, StorageClass

For production storage you need three objects working together:

Access Modes

Most cloud block storage (AWS EBS, GCE PD) only supports RWO. NFS or cloud file systems support RWX.

Reclaim Policies

The Binding Lifecycle

1. You create a PVC specifying storageClass, accessMode, and size.
2. The StorageClass provisioner creates a matching PV (dynamic provisioning).
3. The PVC moves to Bound state — the PV is exclusively reserved for this PVC.
4. You reference the PVC in a Pod spec via persistentVolumeClaim.claimName.
5. kubelet mounts the volume on the node before starting the container.

PV/PVC Binding & CSI Architecture

  +-----------------------+              +------------------------+
  |    PersistentVolume   |              | PersistentVolumeClaim  |
  |  (Cluster-wide Disk)  | <=========>  | (Namespace-scoped Req) |
  +-----------+-----------+   (Binding)  +-----------+------------+
              ^                                      ^
   Dynamic    |                                      | Referenced
   Provision  |                                      | by Pod
              |                                      v
  +-----------+-----------+              +-----------+------------+
  |     StorageClass      |              |          Pod           |
  |  (Provisioner Config) |              |  (cgroups & Namespaces)|
  +-----------+-----------+              +-----------+------------+
              |                                      |
              | CSI gRPC API                         | Kubelet Node
              v                                      v Sync
  +-----------+-----------+              +-----------+------------+
  |  CSI Controller Plugin|              |   Kubelet Volume Mgr   |
  |  - CreateVolume       |              |   - NodeStageVolume    |
  |  - ControllerPublish  |              |   - NodePublishVolume  |
  +-----------------------+              +-----------+------------+
                                                     |
                                                     v
                                         +-----------+------------+
                                         | Linux Host Node        |
                                         | - /dev/sdX (Block Dev) |
                                         | - /var/lib/kubelet/... |
                                         +------------------------+

Under the Hood: The CSI Specification & Mount Lifecycle

When a Pod requesting persistent storage is scheduled, the control plane and node agent orchestrate volume provisioning and mounting via the Container Storage Interface (CSI) gRPC API:

1. PV-PVC Matching and Binding

The pv-controller loop in the kube-controller-manager matches a newly created PVC to a PV. It compares:

• StorageClassName: Must match exactly (or default).
• AccessModes: PV must support all modes requested in PVC.
• Capacity: PV capacity must be greater than or equal to PVC request.
• Selectors: PV labels must match the PVC's matchLabels or matchExpressions.

If no matching PV is found, and the StorageClass has a dynamic provisioner, the controller calls the CSI plugin's CreateVolume gRPC endpoint to provision the block storage in the cloud/infra.

2. Controller Attachment (Attach)

Once bound, the external-attacher controller calls the CSI controller's ControllerPublishVolume gRPC endpoint. This instructs the cloud/storage provider to attach the raw block device (e.g., /dev/xvdf) to the Linux host node where the Pod has been scheduled.

3. Node Staging (Format & Mount)

Inside the target node, the Kubelet's Volume Manager takes over. It executes the CSI node plugin's NodeStageVolume gRPC call, which:

• Checks if the device has a filesystem (e.g., ext4, xfs). If not, it formats the block device.
• Mounts the formatted device to a global staging directory on the node:

/var/lib/kubelet/plugins/kubernetes.io/csi///globalmount

4. Node Publishing (Bind Mount)

Finally, Kubelet executes NodePublishVolume. This performs a Linux bind mount from the global staging directory to the Pod's specific mount directory:

/var/lib/kubelet/pods//volumes/kubernetes.io~csi//mount

Kubelet then configures the container runtime (CRI gRPC API) to launch the container, passing this path as a mount point. The container runtime uses Linux mount namespaces (CLONE_NEWNS) and cgroups to expose this directory as a directory inside the container's isolated filesystem structure.