Observability: Metrics, Logging & Tracing

The Three Pillars of Observability

To understand what is happening inside a distributed system you need three complementary signals:

No single pillar is sufficient on its own. Metrics tell you *something is wrong*. Logs tell you *what happened*. Traces tell you *where in the request chain it went wrong*.

Visualizing Kubernetes Observability Data Flow

+---------------------------------------------------------------------------------+
|                                Kubernetes Node                                  |
|                                                                                 |
|  +--------------------+  (exposes stats)  +----------------------------------+  |
|  | Application Pod    | ----------------> | Kubelet (Port 10250)             |  |
|  | (cgroups v2 limits)|                   | - Stats Summary API: /stats/sum  |  |
|  |                    |                   +-----------------+----------------+  |
|  | - Stdout/Stderr    |                                     |                   |
|  +---------+----------+                                     v (scrapes stats)   |
|            |                                    +-----------+------------+      |
|            | (writes logs)                      | metrics-server         |      |
|            v                                    | (exposes metrics.k8s)  |      |
|  +---------+----------+                         +-----------+------------+      |
|  | /var/log/pods/     |                                     |                   |
|  +---------+----------+                         +-----------+------------+      |
|            ^                                    v (kubectl top)                 |
|            | (inotify / tail)                   +-----------+------------+      |
|  +---------+----------+                         | API Server / kubectl   |      |
|  | Log DaemonSet      |                         +------------------------+      |
|  | (Fluent Bit / Loki)| ----> [ Loki / ES ]                                     |
|  +--------------------+                                                         |
|                                                                                 |
|  +--------------------+  (/metrics endpoint)    +------------------------+      |
|  | Application Pod B  | <---------------------- | Prometheus Server      |      |
|  +--------------------+  (scrapes HTTP/gRPC)    +-----------+------------+      |
|                                                             |                   |
|                                                             v (visualizes)      |
|                                                         [ Grafana ]             |
+---------------------------------------------------------------------------------+

Metrics Stack: Prometheus + Grafana

Prometheus collects metrics using a pull model: it scrapes HTTP /metrics endpoints on a schedule. Every pod that exposes Prometheus-format metrics is automatically collected.

Key components:

• kube-state-metrics: runs as a Deployment, exposes Kubernetes *object state* as metrics — kube_pod_status_phase, kube_deployment_status_replicas_available, etc.
• node-exporter: runs as a DaemonSet (one per node), exposes *node-level hardware metrics* — CPU, memory, disk I/O, network bandwidth
• Prometheus Operator: manages Prometheus configuration via CRDs. The ServiceMonitor CRD lets you declaratively tell Prometheus "scrape all Services with label app=myapp"
• Alertmanager: routes alerts from Prometheus to Slack, PagerDuty, email, etc.

Under the hood, when you apply a ServiceMonitor CRD, the Prometheus Operator controller detects the change, translates the resource selector parameters into raw Prometheus scrape configuration blocks, writes them into the Prometheus configuration file mounted inside the Prometheus container, and triggers a hot-reload of Prometheus via an HTTP POST request to its /-/reload API endpoint, preventing any downtime or server restarts.

Grafana visualizes Prometheus data with pre-built dashboards (cluster overview, node exporter, workload metrics).

kubectl top: Quick Resource Usage

kubectl top requires metrics-server — a lightweight deployment that aggregates CPU/memory usage from the kubelet on each node.

kubectl top nodes          # CPU and memory per node
kubectl top pods -A        # CPU and memory per pod, all namespaces
kubectl top pods --sort-by=memory -n production  # sorted by memory

metrics-server query path:

1. The metrics-server regularly calls the kubelet's Summary API (/stats/summary on port 10250) on every node.
2. The kubelet reads cgroup files from the host Linux kernel (cgroups v2 paths like /sys/fs/cgroup/system.slice/containerd.service/..., reading cpu.stat and memory.current) to calculate container CPU millicores and memory usage.
3. metrics-server serves these data points through the API extension gateway under the metrics.k8s.io API path, which kubectl queries.

metrics-server is the data source for HPA. Prometheus is for historical data and alerting.

Logging Stack

Kubernetes does not provide centralized logging — logs are per-pod and lost when the pod is deleted.

The standard pattern:

1. Apps write to stdout/stderr (never to files)
2. The container runtime (e.g. containerd) redirects stdout/stderr streams to host files at /var/log/pods/__//.log.
3. A DaemonSet (Fluent Bit, Fluentd, or Promtail) runs on every node and tails pod logs using Linux kernel inotify watch events.
4. The DaemonSet queries the local kubelet or API server to append metadata (labels, namespace, annotations) to each log line, before shipping them to a backend like Loki (lightweight, indexes metadata only) or Elasticsearch (full-text indexed, heavier)

Structured logging (JSON format) is strongly preferred:

• Fields are indexable and filterable in any log backend
• Machines can parse and route by field values
• Plain text logs require fragile regex parsing

Distributed Tracing & Context Propagation

OpenTelemetry is the CNCF standard SDK for instrumentation. Add it to your application code to emit trace spans. Spans are collected by an OpenTelemetry Collector and forwarded to a backend:

• Jaeger: open-source trace backend with UI
• Grafana Tempo: Grafana-native trace backend that integrates with Loki

For tracing to work across microservice hops, applications must propagate the trace context. The standard W3C Trace Context defines headers like traceparent (e.g. 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01), which contains the overall Trace ID, Parent Span ID, and flags. When Service A calls Service B, it injects this header into HTTP/gRPC metadata. Service B extracts it to establish the parent-child relationship between spans.