A container that exits is restarted by the kubelet, by default forever. After each exit the kubelet waits twice as long, from 10 seconds up to 5 minutes. That waiting state is CrashLoopBackOff, and the RESTARTS column counts the attempts.
Three questions
kubectl describe pod market-api-7b6c-x2k4p Last State: Terminated
Reason: Error
Exit Code: 2
Restart Count: 6The Last State block answers the first question: how did it die? The exit code says who decided.
| Exit code | Meaning | Look at |
|---|---|---|
| 0 | finished normally; a Deployment restarts it anyway | the command, should this be a Job? |
| 1, 2 | the application failed | logs --previous |
| 127 | command not found | command: and the image |
| 137 | killed by SIGKILL, usually OOMKilled | memory limit |
| 139 | segfault | the application |
The second question is what it said before it died:
kubectl logs market-api-7b6c-x2k4p --previous --tail=20--previous reads the container that just crashed. Without it you read the new attempt, often before it has printed anything.
The third question is whether anyone killed it. OOMKilled as the reason means the kernel did, because the container passed its memory limit. A liveness probe shows up as Events:
Warning Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 404
Normal Killing Container web failed liveness probe, will be restartedThat container may be healthy; the probe is checking a path that does not exist.
Fixing it
An application error is fixed in config or code: the environment variable it wants, the host it cannot reach. OOMKilled is fixed by raising resources.limits.memory or lowering the working set. A probe failure is fixed on the probe first: right path, right port, enough initialDelaySeconds.