Troubleshooting Networking

Fix "connection refused", DNS lookup failures, and NetworkPolicy blocks using kubectl exec, port-forward, and endpoint inspection.

advanced90 minRead → Lab → Quiz → Practice

🧠 Brain Warm-Up

🧠 Brain Warm-Up: A Service has three pods behind it. You curl the Service ClusterIP and get "connection refused". But if you curl one Pod IP directly, it works. Without running any commands, list the three possible root causes. Which one is most likely?

Network Debugging Decision Tree

curl <service-ip>:<port> fails (connection refused / timeout)
       │
       ├── Step 1: Check endpoints
       │   kubectl get endpoints <service-name>
       │   │
       │   ├── ENDPOINTS: <none> ──────────────────► Selector mismatch
       │   │                                          Compare: kubectl get svc -o yaml | grep selector
       │   │                                          vs:       kubectl get pods --show-labels
       │   │                                          Fix:      align selector with pod labels
       │   │
       │   └── ENDPOINTS: 10.244.x.x:8080 ─────────► Endpoints exist, problem is elsewhere
       │
       ├── Step 2: Test Pod directly (bypass Service)
       │   kubectl port-forward pod/<pod-name> 8080:8080
       │   curl localhost:8080
       │   │
       │   ├── WORKS ─────────────────────────────► Service port mapping is wrong
       │   │                                         Check: svc.spec.ports.targetPort vs container port
       │   │
       │   └── FAILS ─────────────────────────────► Pod is not serving on expected port
       │                                             kubectl exec -it <pod> -- ss -tlnp
       │                                             kubectl logs <pod>
       │
       └── Step 3: Check NetworkPolicy
           kubectl get networkpolicy -n <namespace>
           kubectl describe networkpolicy <name>
           Is the pod selected? Does ingress/egress allow this traffic?

DNS Debugging Decision Tree

DNS lookup fails inside a pod
       │
       ├── Step 1: Is CoreDNS running?
       │   kubectl get pods -n kube-system -l k8s-app=kube-dns
       │   kubectl logs -n kube-system -l k8s-app=kube-dns
       │
       ├── Step 2: Test DNS from inside a pod
       │   kubectl exec -it <pod> -- nslookup kubernetes
       │   kubectl exec -it <pod> -- nslookup <service-name>
       │   kubectl exec -it <pod> -- nslookup <service>.<namespace>.svc.cluster.local
       │
       ├── Step 3: Cross-namespace DNS
       │   Short name:  my-db          (only works within same namespace)
       │   Full FQDN:   my-db.backend.svc.cluster.local  (works from any namespace)
       │
       └── Step 4: Check /etc/resolv.conf in the pod
           kubectl exec -it <pod> -- cat /etc/resolv.conf
           # Should show: nameserver 10.96.0.10 (kube-dns ClusterIP)

Service Selectors and Endpoints

A Service finds its Pods via spec.selector. If the selector does not match any Pod labels, kubectl get endpoints shows :

# Check the service selector
kubectl get svc my-svc -o jsonpath='{.spec.selector}'
# Output: {"app":"backend"}

# Check actual pod labels
kubectl get pods --show-labels
# NAME          READY   STATUS    LABELS
# backend-pod   1/1     Running   app=backend-v2,tier=app   ← label mismatch!

# Fix: update either the service selector or the pod labels
kubectl patch svc my-svc --type=merge -p '{"spec":{"selector":{"app":"backend-v2"}}}'

NetworkPolicy — Default-Deny Trap

Kubernetes defaults to allow all. The moment you apply a NetworkPolicy that selects a pod, all traffic not explicitly allowed is denied:

# This policy selects all pods in 'default' namespace
# and allows NO ingress traffic — effectively default-deny
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}    # selects ALL pods in namespace
  policyTypes:
  - Ingress
  # No ingress rules = deny all ingress

To debug: kubectl describe networkpolicy and trace whether the traffic source/destination matches any rule.

Useful Debugging Commands

# Check Service and Endpoints together
kubectl get svc,endpoints my-svc

# Curl a service from inside a debug pod
kubectl run debug --image=curlimages/curl:8.7.1 --rm -it --restart=Never -- curl http://my-svc:80

# DNS lookup
kubectl run debug --image=busybox:1.36 --rm -it --restart=Never -- nslookup my-svc
kubectl run debug --image=busybox:1.36 --rm -it --restart=Never -- nslookup my-svc.default.svc.cluster.local

# Check what port the app is actually listening on
kubectl exec -it <pod> -- ss -tlnp
kubectl exec -it <pod> -- netstat -tlnp  # if available

# Port-forward to bypass Service routing
kubectl port-forward pod/<pod-name> 8080:80

←

Troubleshooting Pods

Troubleshooting Nodes

→