Demystifying CrashLoopBackOff: How to Diagnose and Fix Crashing Kubernetes Pods Using kubectl

Few status messages in cloud computing inspire as much dread as CrashLoopBackOff. You commit your code, the CI/CD pipeline turns green, and your pods deploy—only for kubectl get pods to reveal an ever-increasing restart count and a status that refuses to stay Running.

The name describes exactly what is happening under the hood: your container starts, immediately crashes, and Kubernetes backs off exponentially before attempting to restart it again.

Because the container is caught in a cycle of crashing and waiting, inspecting it using standard tooling often results in empty screens and dead ends. This guide provides a step-by-step diagnostic framework using kubectl logs and kubectl describe to quickly pinpoint and resolve the root cause of any CrashLoopBackOff.

1. Why kubectl logs Fails by Default

The most common mistake engineers make when troubleshooting a CrashLoopBackOff is running:

Bash

kubectl logs <pod-name>

More often than not, this command returns nothing at all or outputs a single generic startup line.

Why Does This Happen?

When a pod is in CrashLoopBackOff, Kubernetes is waiting out an exponential back-off delay (10s, 20s, 40s, up to 5 minutes) before spinning up a replacement container. If you run kubectl logs during this delay or right as a new container is instantiated:

1. The container that actually threw the fatal exception has already been terminated.

2. The new container has not written anything to stdout or stderr yet.

By default, kubectl logs only displays the output of the currently running instance of a container. To find out why your application crashed, you must query the instance that preceded it.

2. Step 1: Inspect the Previous Container Instance

The single most critical flag in your debugging toolkit is --previous (or -p):

Bash

kubectl logs <pod-name> --previous

What It Does

The --previous flag instructs the container runtime (containerd/CRI-O) to pull the termination logs from the last crashed container instance before Kubernetes attempted to restart it.

What to Look For

Scroll to the very bottom of the output. In 80% of application-level bugs, this reveals the exact stack trace or fatal termination event:

• Missing Environment Variables: A null pointer or runtime exception triggered because an expected secret or database connection string was omitted.

• Database / Network Timeouts: The application attempts an immediate connection at boot and panics when downstream services do not answer within the timeout window.

• Port / Permission Bind Failures: The containerized process attempts to bind to a privileged port (e.g., port 80) while running as a non-root security context.

• Syntax / Entrypoint Errors: A malformed start script, missing dependency binary, or bad command passed in the Dockerfile.

3. Step 2: Streaming Logs in Real Time

If your application takes a few seconds or minutes to boot before hitting a panic condition, catching the logs before the crash occurs can be challenging.

In this scenario, stream the standard output in real time using the -f (follow) flag:

Bash

kubectl logs -f <pod-name>

This keeps the session attached to stdout/stderr. As Kubernetes triggers the next restart, you will see the runtime initialization logs stream in line by line until the crash terminates the process.

(Press Ctrl + C to detach from the log stream.)

4. Multi-Container Pod Considerations

If your pod architecture includes sidecars (such as Envoy, Istio proxies, or log shippers), a generic kubectl logs command will fail with an error like:

Plaintext

Error from server (BadRequest): a container name must be specified for pod <pod-name>, choose one of: [app-container istio-proxy]

To direct your logs query to the specific failing application container, supply the -c argument:

Bash

kubectl logs <pod-name> -c <container-name> --previous

5. What If the Logs Are Completely Empty?

If running kubectl logs --previous produces no output whatsoever, your application did not crash on its own.

Instead, the host operating system or the Kubernetes runtime terminated the container abruptly before the process could write a stack trace to standard output. When this happens, application logs are useless; you must consult the platform events.

Step 3: Inspect the Pod Lifecycle and Exit Codes

Run the describe command to read the container state and event stream:

Bash

kubectl describe pod <pod-name>

Inspect two primary sections of the resulting output:

A. The Container Last State Block

Look for the Last State block under Containers:

YAML

Containers:
  my-app:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

• Exit Code 137 (OOMKilled): Out of Memory. The container attempted to allocate more memory than permitted by the limits.memory defined in your deployment.yaml. The Linux kernel Out-Of-Memory (OOM) killer dispatched a SIGKILL (signal 9, where $128 + 9 = 137$) immediately. To fix this, allocate higher memory limits or address runtime memory leaks.

• Exit Code 143 (SIGTERM): The process received a graceful termination signal from Kubernetes that timed out, followed by a SIGKILL. This frequently indicates a failing liveness probe.

• Exit Code 1 or 2: Standard application/runtime fatal error. If logs are absent, verify file permissions and binary execute permissions.

• Exit Code 126 or 127: Command not executable / Command not found. The ENTRYPOINT or CMD path inside the Docker image is invalid.

B. The Events Section

Scroll to the bottom of the kubectl describe output. The event stream logs actions executed by the kubelet on the host node:

• Liveness probe failed: HTTP probe failed with statuscode: 500: If your liveness probe checks a health endpoint (/healthz) that fails repeatedly during startup, the kubelet will repeatedly kill and restart the container, trapping it in a CrashLoopBackOff. Increase initialDelaySeconds or optimize application boot times.

• Failed to attach/mount volumes: The pod cannot start because a dependent Secret, ConfigMap, or Persistent Volume Claim (PVC) does not exist or is not accessible.

Summary Diagnostic Workflow

When facing an elusive CrashLoopBackOff, follow this structured sequence: