Secrets

Store and consume sensitive data like passwords, tokens, and TLS certificates safely.

intermediate60 minRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: What is the default encryption state of Kubernetes Secrets in etcd, and what low-level mechanism ensures they do not get written to the physical storage disk of the worker nodes?

What Are Secrets?

Secrets are designed to store sensitive configuration data such as database passwords, API tokens, private TLS keys, and docker registry credentials.

While they behave similarly to ConfigMaps, their lifecycle, storage, and access paths are engineered to prevent exposure.

Visualizing Secrets Security Flow

                  +--------------------------------+
                  |  kubectl apply -f secret.yaml  | (Base64 Encoded Payload)
                  +--------------------------------+
                                  |
                                  v
                  +--------------------------------+
                  |      kube-apiserver            |
                  |  (Optionally encrypts via KMS) |
                  +--------------------------------+
                                  |
                                  v
                  +--------------------------------+
                  |           etcd                 | (Stored in Raft consensus:
                  |                                |  Plaintext OR KMS Encrypted)
                  +--------------------------------+
                                  |
                   (mTLS / API Server watches)
                                  |
                                  v
                  +--------------------------------+
                  |            Kubelet             |
                  +--------------------------------+
                                  |
               (Mounts payload into worker node memory)
                                  v
                  +--------------------------------+
                  |      tmpfs (RAM disk)          | (Never written to worker node
                  |  /var/lib/kubelet/pods/...     |  physical storage disk)
                  +--------------------------------+

Base64 vs Encryption at Rest

A common point of confusion is that Secrets are "encrypted" in the manifest. Base64 is a serialization encoding scheme, not encryption. It exists to allow binary files (like certificates) to be safely encoded into YAML strings without breaking parser syntax.

# To decode a secret from YAML:
echo "czNjcjN0" | base64 --decode  # outputs: s3cr3t

To secure Secrets at rest in the backend database (etcd, which uses Raft consensus to replicate state), you must configure EncryptionConfiguration on the kube-apiserver.

Without this configuration, keys and values are stored in etcd as plaintext. The API server supports multiple providers for encryption:

Local Providers: aescbc (AES-CBC), secretbox (XSalsa20 and Poly1305).
KMS Provider: Integrates with external Key Management Services (AWS KMS, GCP KMS, HashiCorp Vault) via a local gRPC plugin. The KMS provider uses envelope encryption: a local Key Encryption Key (KEK) encrypts the data, and the KMS encrypts the KEK.

Host-Level Security: tmpfs & Env Vars

To secure secrets on the physical worker nodes where Pods run, Kubernetes avoids writing them to disk:

tmpfs Mounts: When a Secret is mounted as a volume, the kubelet creates a tmpfs (RAM disk) volume inside the host memory. The secret files are written directly into RAM. If the node is powered down or rebooted, the secret data vanishes instantly from host memory.
Environment Variables: While convenient, using secretKeyRef to inject secrets as environment variables is discouraged for high-security workloads. Environment variables are visible in plaintext to anyone with container process-inspection rights via /proc//environ and are frequently dumped into application log aggregators during crashes.

Secret Types

Secret Type	Required Data Keys	Intended Usage
`Opaque`	User-defined	Default type for generic key-value pairs (e.g. database credentials).
`kubernetes.io/tls`	`tls.crt`, `tls.key`	Stores a public certificate and matching private key. Used by Ingress controllers to terminate TLS.
`kubernetes.io/dockerconfigjson`	`.dockerconfigjson`	Stores registry authentication credentials (base64 JSON config) used by kubelet to pull images from private registries via `imagePullSecrets`.
`kubernetes.io/service-account-token`	`token`, `ca.crt`	Contains a JWT token representing the ServiceAccount. Historically auto-created, now primarily generated dynamically via the TokenRequest API for security.

Best Practices

Never commit Secret YAML to git — even base64-encoded values are easily decoded
Use external secret stores in production: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager
Enable EncryptionConfiguration on the API server for encryption at rest
Scope RBAC tightly — only the service accounts that need a Secret should be able to read it
Prefer volume mounts over env vars — env vars can be leaked in crash dumps and logs

Secrets are NOT more secure than ConfigMaps by default. Security comes from etcd encryption + RBAC, not the Secret object type itself.