Things break in Kubernetes the way they break anywhere: one piece at a time. The trick is knowing which piece. Every problem you will meet in this chapter lives in one of five layers.
the container itself: image, command, crash, probes, resources
Deployment, ReplicaSet, Job: paused, scaled to zero, blocked by quota
selector, ports, endpoints, DNS, NetworkPolicy
ConfigMaps, Secrets, ServiceAccounts, RBAC
nodes, scheduling, storage, capacity
From symptom to layer
| Symptom | Start at |
|---|---|
Pod Pending | cluster: describe pod, the FailedScheduling event |
Pod CrashLoopBackOff, Error | Pod: logs --previous, exit code |
Pod ErrImagePull, ImagePullBackOff | Pod: image name and tag |
Pod CreateContainerConfigError | config: the ConfigMap or Secret it references |
Pod Running but 0/1 | Pod: readiness probe |
| Pod fine, no traffic | Service: selector, ports, endpoints |
Deployment 0/3, no Pods | controller: ReplicaSet Events, quota, paused |
Four commands
sh
kubectl get pods -o wide
kubectl describe pod market-api-7b6c-x2k4p
kubectl logs market-api-7b6c-x2k4p --previous
kubectl events --for pod/market-api-7b6c-x2k4pget tells you which layer. describe tells you why, in the Events at the bottom and the container state in the middle. logs is the application's own account. events for a Deployment or ReplicaSet covers the controller layer.
Change one thing
Form a guess, change exactly one thing, and run the same get again. If the symptom moved, you learned something. If it did not, undo the change before trying the next one. A fix made of three simultaneous edits leaves two of them unexplained in your manifests forever.