Kubernetes Troubleshooting Labs: CrashLoopBackOff, ImagePullBackOff and Stuck Rollouts

Kubernetes troubleshooting is a skill you build by fixing broken things, and production is an expensive place to get your first reps. This guide covers the failures you'll meet first: CrashLoopBackOff, ImagePullBackOff, a rollout that hangs, a rollout that never starts, and manifests that won't apply. For each one it shows what the failure looks like, which command tells you why, and how to fix it so the fix sticks. Each section ends with a free lab where the failure is already set up on a cluster in your browser, and the lab checks the cluster's state when you're done.

One loop for every failure

Most investigations follow the same four commands, from the summary down to the detail:

kubectl get pods -n shop                          # STATUS, READY, RESTARTS: what kind of failure
kubectl describe pod <pod> -n shop                # State, Last State, Events: what Kubernetes saw
kubectl logs <pod> -n shop --previous             # what the application said before it died
kubectl get events -n shop --sort-by=.lastTimestamp   # everything recent in the namespace, in order

The STATUS column tells you where to look next:

CrashLoopBackOff: the container keeps exiting

$ kubectl get pods -n shop
NAME         READY   STATUS             RESTARTS      AGE
market-api   0/1     CrashLoopBackOff   5 (38s ago)   4m
market-web   1/1     Running            0             4m

CrashLoopBackOff isn't an error in itself. It's the kubelet's waiting state between restarts. Each time the container exits, the kubelet waits longer before starting it again: 10 seconds, then 20, 40, and so on up to five minutes. The delay resets once a container has run for ten minutes without exiting. The question is always why the container exits, and the application usually says so on its way out.

kubectl logs shows the current container, which may have just restarted and printed nothing useful yet. --previous shows the one that crashed:

$ kubectl logs market-api -n shop --previous
2026-10-17T09:12:03Z INFO  market-api starting (version 1.4.2)
2026-10-17T09:12:03Z INFO  reading configuration from environment
2026-10-17T09:12:04Z ERROR MONSTER_DB_URL is not set
panic: cannot connect to MONSTER_DB_URL

When the logs are empty, describe tells you how the container ended:

$ kubectl describe pod market-api -n shop
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
...
Events:
  Type     Reason   Age                From     Message
  ----     ------   ----               ----     -------
  Warning  BackOff  12s (x21 over 4m)  kubelet  Back-off restarting failed container market-api in pod market-api_shop(...)

The exit code points you at the cause:

• 1 or 2: the application failed and chose the code. Missing configuration, a dependency it can't reach, a bad flag. The logs have the details.
• 137 with Reason OOMKilled: the container used more memory than resources.limits.memory. Raise the limit or fix the leak. Restarting won't help.
• 137 without OOMKilled: something sent SIGKILL, often a liveness probe that kept failing until the kubelet killed the container. Look for Unhealthy events.
• 0: the process finished successfully. That's correct for a Job, and wrong for a Deployment: the container needs a process that stays in the foreground.

Fix the owner, not the Pod. If the Pod belongs to a Deployment, kubectl edit pod changes are lost the next time the Pod is replaced, and most Pod fields can't be edited anyway. Change the Deployment (kubectl set env deploy/market-api MONSTER_DB_URL=..., or edit the manifest and apply it), and it rolls out new Pods with the fix.

ImagePullBackOff: the node can't get the image

$ kubectl get pods -n shop
NAME         READY   STATUS             RESTARTS   AGE
market-web   0/1     ImagePullBackOff   0          2m

ErrImagePull is the failed attempt; ImagePullBackOff is the wait before the next attempt, with the same growing delay as crash loops. The reason is always in the Pod's events:

$ kubectl describe pod market-web -n shop
...
Events:
  Type     Reason   Age               From     Message
  ----     ------   ----              ----     -------
  Normal   Pulling  95s (x4 over 2m)  kubelet  Pulling image "nginx:1.99-nope"
  Warning  Failed   94s (x4 over 2m)  kubelet  Failed to pull image "nginx:1.99-nope": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.99-nope": failed to resolve reference "docker.io/library/nginx:1.99-nope": docker.io/library/nginx:1.99-nope: not found
  Warning  Failed   94s (x4 over 2m)  kubelet  Error: ErrImagePull
  Normal   BackOff  80s (x6 over 2m)  kubelet  Back-off pulling image "nginx:1.99-nope"

Read the end of the Failed message. The four causes you'll see most:

• not found: the tag or repository doesn't exist. Usually a typo, or a tag your CI never pushed.
• 401 Unauthorized, pull access denied: the registry is private and the Pod has no credentials. Create a docker-registry Secret and reference it in imagePullSecrets, on the Pod spec or on its ServiceAccount.
• 429 Too Many Requests: the registry is rate-limiting your nodes. Common with anonymous Docker Hub pulls from a busy cluster.
• no match for platform in manifest: the image wasn't built for the node's CPU architecture, for example an amd64-only image on arm64 nodes.

For a Pod you created directly, the image is one of the few fields you can change in place:

kubectl set image pod/market-web market-web=nginx:1.27 -n shop

For a ReplicaSet there's a catch that surprises people: changing its template doesn't touch the Pods it already has. The broken Pods stay broken until they're deleted and the ReplicaSet replaces them from the corrected template. A Deployment is what does that replacement for you, which is one reason you rarely create ReplicaSets yourself.

A rollout that hangs on a bad image

Now the same bad image, through a Deployment with three replicas:

$ kubectl set image deploy/market-web nginx=nginx:1.99-nope -n shop
deployment.apps/market-web image updated

$ kubectl get pods -n shop -l app=market-web
NAME                          READY   STATUS             RESTARTS   AGE
market-web-7c5ddbdf54-2xk9p   1/1     Running            0          6m
market-web-7c5ddbdf54-8hvzq   1/1     Running            0          6m
market-web-7c5ddbdf54-qm4tn   1/1     Running            0          6m
market-web-6f8b9d7c48-x7wlp   0/1     ImagePullBackOff   0          40s

$ kubectl rollout status deploy/market-web -n shop
Waiting for deployment "market-web" rollout to finish: 1 out of 3 new replicas have been updated...

Notice what didn't happen: no old Pod was removed, and the service kept running. A rolling update defaults to maxSurge: 25% and maxUnavailable: 25%. With three replicas, those round to one extra Pod and zero unavailable ones. So the Deployment creates one new Pod and waits for it to become ready before it removes an old one. It never becomes ready, so the rollout stops there.

It stays like that. After progressDeadlineSeconds (600 by default), the Deployment's Progressing condition turns False with reason ProgressDeadlineExceeded, and rollout status exits with an error. Kubernetes does not roll back by itself. You do:

kubectl rollout undo deploy/market-web -n shop          # back to the previous revision
kubectl rollout history deploy/market-web -n shop       # the revisions it remembers

rollout undo puts the previous Pod template back, and the stuck Pod goes away. Set the kubernetes.io/change-cause annotation when you change a Deployment, and rollout history shows why each revision exists instead of <none>.

A rollout that never starts

The opposite case is quieter: you change the image, nothing fails, and nothing changes either.

$ kubectl set image deploy/market-web nginx=nginx:1.26 -n shop
deployment.apps/market-web image updated

$ kubectl get deploy market-web -n shop
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
market-web   3/3     0            3           20m

UP-TO-DATE is 0: none of the Pods run the new template, and none are being created. The Deployment has been paused, which is what kubectl rollout pause is for: batching several changes into one rollout. describe says so:

$ kubectl describe deploy market-web -n shop | grep -A3 Conditions
Conditions:
  Type           Status   Reason
  ----           ------   ------
  Progressing    Unknown  DeploymentPaused

kubectl rollout resume deploy/market-web starts the rollout. The same quiet pattern has two more causes worth checking when "nothing is happening": a Deployment scaled to replicas: 0, and a CronJob with suspend: true, which keeps its schedule but never creates Jobs. Nothing in kubectl get pods will point at any of them, because there's no Pod to look at.

Manifests that won't apply

Some failures never reach the cluster. kubectl apply rejects the file, and the error tells you which kind of mistake it is:

$ kubectl apply -f api.yaml
error: resource mapping not found for name: "api" namespace: "" from "api.yaml": no matches for kind "Deploymnet" in version "apps/v1"
ensure CRDs are installed first

No matches for kind means the API server doesn't know that kind at that apiVersion. Either the kind is misspelled, as here, or the manifest uses an API version that has been removed, such as extensions/v1beta1 for Deployments. kubectl api-resources lists what your cluster serves.

$ kubectl apply -f worker.yaml
The Deployment "worker" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"wrk"}: `selector` does not match template `labels`

selector does not match template labels means spec.selector.matchLabels would never select the Pods the template creates. Make them identical. Remember that a Deployment's selector can't be changed after creation, so on an existing Deployment you change the template's labels or recreate it.

$ kubectl apply -f db.yaml
error: error parsing db.yaml: error converting YAML to JSON: yaml: line 14: did not find expected key

A YAML parse error comes with a line number. The mistake is often on the line before it: inconsistent indentation, a tab character, or a list item that's missing its -.

Then there's the mistake that applies cleanly: a Service whose selector matches no Pods. Nothing errors, and requests fail later. That one gets its own walkthrough.

Put it together

Real incidents rarely have one cause, and fixing the first one often changes the symptom instead of ending it. The best practice is a namespace where several things are broken at once and you have to decide what to look at first. This challenge has a Deployment stuck on a bad image, a worker scaled to zero, an API crash-looping on a placeholder image, a failing Job and a CronJob on the wrong schedule:

Work through it with the loop from the top of this page. Look at the summary, read the events and logs for the one that's failing, fix the object that owns the Pods, then check again before you move to the next fault.