Cloud Explainers

How to Fix a Kubernetes Pod Stuck in CrashLoopBackOff

Kubernetes8 min video

CrashLoopBackOff means your container's program keeps exiting, and Kubernetes waits longer before each restart. On a real Kubernetes cluster we read the last run with kubectl logs --previous and kubectl describe, and fix four common causes: a missing environment variable, a memory limit that gets the container OOM killed, a liveness probe on the wrong port, and a one-off script run as a Deployment instead of a Job.

It runs on the laptop

Your pod is stuck in CrashLoopBackOff, and the restart count keeps climbing. Most people guess the image is broken. Jim's isn't: his lane booking board for Riverside Pool runs fine on his laptop. In Kubernetes, it restarts again and again. The pod isn't stuck, exactly. It's working very hard at failing. There are four common reasons a pod does this. In one of them, nothing is even wrong with the program. And two commands tell you which one you have.

Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
lanes-f7669455-97s8v  0/1  Error  4 (2m ago)  2m43s
lanes-f7669455-p2pwk  0/1  CrashLoopBackOff  4 (68s ago)  2m43s
Jim's laptop
docker run -d --rm --name board -p 8080:8080 -e BOOKING_KEY="$(cat booking.key)" lanes:1.031f07c16f6085802f1400583c42aa62c6103f8b0b1c52f21e141d97336692b72
Jim's laptop
curl -s localhost:8080Riverside Pool lanes 1.0
Lane 1: early swim
Lane 2: lessons
Lane 3: open
served by 31f07c16f608

Missing variable: read the last run

The usual fix starts by reading the run that crashed. kubectl, the Kubernetes command line tool, has logs, and with dash dash previous it shows what the program printed the last time, before it died. Here, it's a Python KeyError: the program looked for a setting called BOOKING_KEY, and it wasn't there. The app reads that key from an environment variable. On the laptop, Jim passed it with dash e. The deployment, the file that tells Kubernetes how to run his app, never set it, so the program exits at once. The second command, kubectl describe pod, shows that exit; the grep part just keeps the Last State lines: reason Error, exit code 1, a program's way of saying it stopped on an error.

Jim's laptop
kubectl logs lanes-f7669455-p2pwk --previousTraceback (most recent call last):
  File "/app/app.py", line 7, in <module>
  KEY = os.environ["BOOKING_KEY"]  # the key the front desk's till checks
  ~~~~~~~~~~^^^^^^^^^^^^^^^
  File "<frozen os>", line 717, in __getitem__
KeyError: 'BOOKING_KEY'
Jim's laptop
kubectl describe pod lanes-f7669455-p2pwk | grep -A4 'Last State'  Last State:  Terminated
  Reason:  Error
  Exit Code:  1
  Started:  Sat, 21 Nov 2026 16:05:13 +0000
  Finished:  Sat, 21 Nov 2026 16:05:14 +0000

So Jim stores the key in the cluster as a secret, Kubernetes' place for passwords and keys, and five lines in the deployment hand it to the container as BOOKING_KEY. kubectl apply sends the changed file, and the rollout finishes. Jim runs two copies of the app, and both are running with zero restarts.

Jim's laptop
kubectl create secret generic booking --from-file=key=booking.keysecret/booking created
Jim's laptop
grep -A5 env: deployment.yaml  env:
  - name: BOOKING_KEY
  valueFrom:
  secretKeyRef:
  name: booking
  key: key
Jim's laptop
kubectl apply -f deployment.yamldeployment.apps/lanes configured
Jim's laptop
kubectl rollout status deployment lanes --timeout=120sWaiting for deployment "lanes" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "lanes" rollout to finish: 1 old replicas are pending termination...
deployment "lanes" successfully rolled out
Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
lanes-64976b6884-8ptql  1/1  Running  0  2s
lanes-64976b6884-97d5l  1/1  Running  0  2s

What CrashLoopBackOff means

That was the easy case: the log named the problem. In the next three, it won't, so here's what the loop really is. Each machine in your cluster, called a node, runs a program called the kubelet, and the kubelet starts your container. When your program exits, the kubelet notes the reason and the exit code, waits, and starts it again. Each wait doubles. CrashLoopBackOff is the name for that wait. Right after a crash, the status says Error for a moment instead. Same loop. So the fix is never in the restart. It's in why the program exits.

CrashLoopBackOff: a loop, not a stuck pod: diagram of kubelet, Container, Program exits, Wait, One pod, same node

Why does that matter? We timed one crashing pod: the first restart came at once. Then the gaps grew: 15 seconds, then 23, then 60, roughly doubling each time, until they reached five minutes, and stayed there. So if your fix is outside the pod, like a missing database coming back, the pod may wait five minutes before it even tries again. Applying a changed deployment is different: it starts new pods right away.

Seconds between starts, one crashing pod: chart of wait 1: 15 s, wait 2: 23 s, wait 3: 60 s, wait 4: 82 s, wait 5: 166 s

The Kubernetes documentation gives the same rule: start at ten seconds, double each time, and stop at three hundred, those five minutes. Ours ran a little later than that rule, never sooner. And once a container runs for ten minutes without a problem, the count starts over.

What CrashLoopBackOff means: browser screenshot from the video

Think of a pull-start lawn mower that starts, coughs, and dies. Pulling the cord again is the restart. Resting your arm a bit longer each time is the back-off. But no number of pulls fixes it. You have to find out why it dies: no fuel, a choke set wrong, or a job that was already done.

Like a pull-start lawn mower: comparison table of Lawn mower vs Your pod, Starts, Dies, Again, Rest, Fix

Out of memory: no error in the log

What if the log has no error at all? Jim's version one point one keeps every member's photo in memory, so the front desk sees it at once. Its pods crash too. The last run printed loading member photos, and then nothing. No error, no traceback. Guess first: what stopped it? Memory. The status column says it this time: OOMKilled, short for out of memory, killed. And describe adds exit code 137.

Jim's laptop
kubectl logs lanes-85bd8d6ff8-bzqz6 --previousloading member photos into memory...
Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
lanes-85bd8d6ff8-84gtl  0/1  CrashLoopBackOff  4 (70s ago)  2m28s
lanes-85bd8d6ff8-bzqz6  0/1  OOMKilled  4 (113s ago)  2m28s
Jim's laptop
kubectl describe pod lanes-85bd8d6ff8-bzqz6 | grep -A4 'Last State'  Last State:  Terminated
  Reason:  OOMKilled
  Exit Code:  137
  Started:  Sat, 21 Nov 2026 16:07:52 +0000
  Finished:  Sat, 21 Nov 2026 16:07:53 +0000

Jim's deployment set a memory limit of 64Mi, sixty-four mebibytes, about sixty-seven megabytes. The photos alone are a hundred and fifty. When a container goes over its limit, the kernel, the core of Linux, kills it on the spot, so the program never gets to print an error. It sends signal 9, a force quit no program can catch. When a signal kills a program, Linux reports 128 plus the signal number. 128 plus 9 is 137. Jim raises the limit to 256Mi in the file, applies it, and both pods run.

Jim's laptop
grep -A2 limits: deployment.yaml  limits:
  memory: 64Mi
Jim's laptop
kubectl apply -f deployment.yamldeployment.apps/lanes configured
Jim's laptop
kubectl rollout status deployment lanes --timeout=180sWaiting for deployment "lanes" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "lanes" rollout to finish: 1 old replicas are pending termination...
deployment "lanes" successfully rolled out

Liveness probe on the wrong port

Now a crash where the app is perfectly healthy. Jim adds a liveness probe from a guide: the kubelet asks the app every five seconds if it's alive, and restarts it when it doesn't answer. The pods crash loop again, but the last log just says listening on port 8080. At the bottom of kubectl describe, a list called Events tells the real story: Liveness probe failed, connection refused on port 8000. The guide's app used 8000. Jim's listens on 8080.

Jim's laptop
grep -A4 livenessProbe: deployment.yaml  livenessProbe:
  httpGet:
  path: /healthz
  port: 8000
  periodSeconds: 5
Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
lanes-865db9459d-8qwkg  0/1  CrashLoopBackOff  5 (1s ago)  4m31s
lanes-865db9459d-ccjhz  1/1  Running  5 (46s ago)  4m31s
Jim's laptop
kubectl logs lanes-865db9459d-8qwkg --previousloading member photos into memory...
loaded 150 MB of member photos
lane board 1.1 listening on port 8080
Jim's laptop
kubectl describe pod lanes-865db9459d-8qwkg | grep -i liveness  Liveness:  http-get http://:8000/healthz delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3
  Warning  Unhealthy  31s (x18 over 4m26s)  kubelet  spec.containers{lanes}: Liveness probe failed: Get "http://10.244.1.11:8000/healthz": dial tcp 10.244.1.11:8000: connect: connection refused
  Normal  Killing  31s (x6 over 4m16s)  kubelet  spec.containers{lanes}: Container lanes failed liveness probe, will be restarted

Each of those runs lasted 45 seconds: three missed checks, then the kubelet politely asks the program to stop, and gives it thirty seconds. Python ignores that request when it's the main program in a container, so the kubelet kills it with signal 9: exit code 137 again. But the reason says Error, not OOMKilled. Read the reason, not just the number.

Jim's laptop
kubectl describe pod lanes-865db9459d-8qwkg | grep -A4 'Last State'  Last State:  Terminated
  Reason:  Error
  Exit Code:  137
  Started:  Sat, 21 Nov 2026 16:13:01 +0000
  Finished:  Sat, 21 Nov 2026 16:13:46 +0000

One number fixes it: the probe asks port 8080, where the app really listens.

Probe the port the app listens on: code change of deployment.yaml, port: 8080 (the app's real port)

Apply the fixed file, wait a minute for the checks to run, and the restart count stays at zero.

Jim's laptop
kubectl apply -f deployment.yamldeployment.apps/lanes configured
Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
lanes-6d8775bb8b-fpsk6  1/1  Running  0  43s
lanes-6d8775bb8b-w59vz  1/1  Running  0  43s

A script that finishes: use a Job

Last, a crash loop where nothing goes wrong at all. Every night, a small script reads the day's new members into the bookings, then finishes. Jim runs it as a second deployment, and it crash loops. The log ends with done. Describe says reason Completed, exit code zero: success.

Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
import-8776d49f4-xh575  0/1  CrashLoopBackOff  4 (78s ago)  2m47s
lanes-6d8775bb8b-fpsk6  1/1  Running  0  3m30s
lanes-6d8775bb8b-w59vz  1/1  Running  0  3m30s
Jim's laptop
kubectl logs import-8776d49f4-xh575 --previousimporting today's new members...
imported 37 members, done
Jim's laptop
kubectl describe pod import-8776d49f4-xh575 | grep -A4 'Last State'  Last State:  Terminated
  Reason:  Completed
  Exit Code:  0
  Started:  Sat, 21 Nov 2026 16:15:57 +0000
  Finished:  Sat, 21 Nov 2026 16:15:59 +0000

Why restart a success? Because a deployment exists to keep a program running, so any exit counts as a crash. For something that should run once and finish, Kubernetes has a Job. It's the mower with the lawn already cut: pulling the cord again helps nobody. A deployment can't be turned into a Job in place, so Jim deletes it first. The file's first two lines now say kind Job. He applies the same script as a Job, and kubectl get jobs shows complete: one of one runs finished. Its pod says Completed, and stays that way. To run it every night, the same idea becomes a CronJob: a Job on a schedule.

Jim's laptop
kubectl get podsNAME  READY  STATUS  RESTARTS  AGE
import-8776d49f4-xh575  0/1  CrashLoopBackOff  4 (78s ago)  2m47s
lanes-6d8775bb8b-fpsk6  1/1  Running  0  3m30s
lanes-6d8775bb8b-w59vz  1/1  Running  0  3m30s
Jim's laptop
kubectl delete deployment importdeployment.apps "import" deleted from default namespace
Jim's laptop
head -n 2 import.yamlapiVersion: batch/v1
kind: Job
Jim's laptop
kubectl apply -f import.yamljob.batch/import created
Jim's laptop
kubectl get jobsNAME  STATUS  COMPLETIONS  DURATION  AGE
import  Complete  1/1  4s  5s

Recap

Read the last run: logs dash dash previous, then describe. An error in the log: fix what it names. OOMKilled: raise the memory limit. Liveness probe failed: fix the probe. Completed: make it a Job.

CrashLoopBackOff: read the last run: diagram of Error, exit 1, OOMKilled, exit 137, Liveness probe failed, Completed, exit 0

If your restart count finally stopped climbing, subscribing gets you more fixes tested on a real cluster.

Every crash in this video came from a real cluster, and every command and the lab files are linked in the description. Thanks for watching Cloud Explainers.

Sources

Demos: a container on our own Mac standing in for a laptop, running a real kind cluster; the swimming pool is made up. Every crash was the real cluster's. Watch the full video on YouTube.

More videos

Python8 min

How to Fix Python ModuleNotFoundError: No Module Named

Go (Golang)5 min

Golang Pointers in 5 Minutes

Homelab & self-hosting9 min

How to Fix a Homelab ESXi Host Not Responding

PostgreSQL24 min

PostgreSQL: Full Tutorial for Beginners

Load balancing & Nginx9 min

How a Load Balancer Works with HAProxy

Python9 min

How Python Virtual Environments Work

Homelab & self-hosting9 min

How to Fix Cloudflare Tunnel Errors in a Homelab

APIs9 min

How to Fix API CORS Errors: No 'Access-Control-Allow-Origin'

Auth (OAuth, JWT)9 min

How Authenticator App Login Codes Work (TOTP)

Git22 min

Git and GitHub: Full Tutorial for Beginners

Homelab & self-hosting10 min

How a Homelab Hypervisor Works (VMware ESXi)

Kubernetes9 min

How to Fix Kubernetes ImagePullBackOff Errors

DNS8 min

How to Fix DNS 'Temporary Failure in Name Resolution' on Linux

AWS8 min

How to Fix AWS CLI 'Unable to Locate Credentials'

Homelab & self-hosting10 min

Proxmox vs ESXi for Your Homelab: When Each One Wins

PostgreSQL10 min

PostgreSQL UUID vs Auto Increment: Which Primary Key Should You Use?

Kubernetes21 min

Kubernetes: Full Tutorial for Beginners

Networking9 min

How TCP Sockets Work

PostgreSQL7 min

How to Fix PostgreSQL 'Could Not Connect to Server'

Claude & the Claude Developer Platform9 min

Claude Code Hooks Will Finally Make Sense After This

DevOps & CI/CD9 min

DevOps with Ansible Will Finally Make Sense After This

Networking9 min

How SNMP Network Monitoring Works

Git8 min

How to Fix GitHub 'Permission Denied (publickey)' Errors

Docker21 min

Docker: Full Tutorial for Beginners

PostgreSQL7 min

How to Fix PostgreSQL 'Password Authentication Failed'

Linux8 min

Why One Linux Server Handles More Users Than You Think

Networking7 min

How to Fix Network 'Connection Refused' Errors

DevOps & CI/CD9 min

DevOps Pipelines: Jenkins vs GitHub Actions

Linux11 min

Linux Nginx Reverse Proxy: Full Tutorial for Beginners

Networking11 min

Every Networking Term a Beginner Needs in 12 Minutes

Docker6 min

How to Fix Docker 'Permission Denied' on docker.sock

Git7 min

How to Fix Git 'Updates Were Rejected' When You Push

Linux7 min

How to Fix nginx 502 Bad Gateway Errors on Linux

Linux8 min

How to Fix Linux apt 'Could Not Get Lock' Errors (Ubuntu)

Networking9 min

How Email Works on the Internet (SMTP) in 10 Minutes

Python8 min

The Python Roadmap for Beginners

Go (Golang)21 min

Golang REST API with Gin: Full Tutorial for Beginners

Networking10 min

How a WireGuard VPN Network Works

Linux9 min

Linux Makefiles Will Finally Make Sense After This

Auth (OAuth, JWT)9 min

SAML vs OAuth vs OpenID Connect: What's the Difference?

Python7 min

How to Fix Python cron Jobs That Never Run

Linux22 min

Linux Bash Scripting: Full Tutorial for Beginners

Networking10 min

How PXE Network Boot Works

Active Directory9 min

How Active Directory Works

Linux10 min

How to Fix a Linux systemd Service That Won't Start

Linux9 min

Linux Logs: syslog vs journald

Auth (OAuth, JWT)9 min

JWT vs Session Cookies: Which One Should You Use?

Docker10 min

Docker vs Docker Compose: Which One Should You Use?

Docker9 min

How to Fix Docker Compose Containers That Can't Talk to Each Other

Django (Python web framework)22 min

Django Full Tutorial for Beginners

Linux9 min

How Linux systemd Works

Networking8 min

How to Fix 'Windows Cannot Access' Network Share Errors (SMB)

APIs10 min

REST vs gRPC: Which One Should You Use?

Certificates & PKI (ADCS)9 min

How to Fix SSL Certificate Chain Errors (Missing Intermediate)

Python9 min

Every Python Concept for Beginners in 10 Minutes

Databases & SQL9 min

Every SQL Join Type in 10 Minutes

OpenRouter (one API for many AI models)9 min

Why OpenRouter Free Models Hit Their Limit So Fast

Gemini API & Google AI Studio9 min

Why Your Gemini API Free Tier Runs Out So Fast

OpenRouter (one API for many AI models)10 min

OpenRouter Will Finally Make Sense After This

Caching & Redis10 min

How Redis Caching Works

Gemini API & Google AI Studio10 min

Gemini API vs Vertex AI: Which One Should You Use?

Gemini API & Google AI Studio8 min

How to Fix a Leaked Gemini API Key

Go (Golang)9 min

Why Golang Uses Goroutines Instead of Threads

Gemini API & Google AI Studio9 min

How to Fix Gemini API Error 429: Resource Exhausted

Claude & the Claude Developer Platform10 min

Claude Token Pricing Will Finally Make Sense After This

Python11 min

Why Python Web Scraping Returns Nothing (and the Fix)

Python10 min

Python uv vs pip: Which One Should You Use?

Python22 min

Python Automation: Full Tutorial for Beginners

Auth (OAuth, JWT)9 min

How OAuth Login Works

Gemini API & Google AI Studio10 min

Google AI Studio Will Finally Make Sense After This

Docker10 min

What Happens When You Run a Docker Container

Claude & the Claude Developer Platform8 min

How to Fix Claude Code Billing Your API Key Instead of Your Plan

Auth (OAuth, JWT)9 min

Why Passkeys Can't Be Phished Like Passwords

PowerShell10 min

PowerShell vs Bash: Which One Should You Learn?

Linux10 min

The Linux Roadmap for Beginners

Django (Python web framework)9 min

Django vs FastAPI: Which One Should You Use?

Databases & SQL10 min

Why SQLite Is the Most Used Database in the World

Databases & SQL11 min

How a NoSQL Database Like MongoDB Works

Python11 min

How to Fix Python pip error: externally-managed-environment

Databases & SQL9 min

How to Fix MySQL Too Many Connections Database Errors

Go (Golang)9 min

Golang Concurrency Will Finally Make Sense After This

AWS10 min

How Amazon S3 Works

Kubernetes10 min

How Kubernetes Works

Git10 min

How Git Works

Docker10 min

How to Fix Docker Port Already Allocated Errors

PostgreSQL10 min

How PostgreSQL Works

Git8 min

How to Fix Git Merge Conflicts

Linux9 min

How to Fix No Space Left on Device in Linux

Networking10 min

How the Internet Works

Databases & SQL9 min

SQL vs NoSQL: When Each One Wins

Auth (OAuth, JWT)8 min

How to Fix JWT Token Expired Errors (401 Unauthorized)

Claude & the Claude Developer Platform9 min

Claude API vs Claude Pro: Why Your Subscription Doesn't Include It

AI agent skills9 min

Why AI Agent Skills Beat Giant Prompts

Linux10 min

Linux Cron Jobs Will Finally Make Sense After This

Cybersecurity10 min

Security Logs Will Finally Make Sense After This

Cybersecurity8 min

How to Fix a Security Leak: An API Key Pushed to GitHub

DNS9 min

DNS Will Finally Make Sense After This

Cybersecurity9 min

Never Ignore a Security Update

Auth (OAuth, JWT)9 min

Two-Factor Authentication Will Finally Make Sense

Networking8 min

Never Expose a Database to the Internet

Cybersecurity10 min

SSH Keys vs Passwords: Which Is Safer for Security?

Linux10 min

The 5 Levels of Linux Boot: From Power On to Login

Networking12 min

The 5 Levels of Network Firewalls

Cybersecurity10 min

Password Security: Why Hackers Crack Weak Passwords

Linux9 min

Why Your Linux Server Runs Out of Memory

Networking10 min

HTTP vs HTTPS: What Actually Changes on Your Network

Linux12 min

Every Linux File Permission You Need to Know

Linux10 min

Linux Processes vs Threads: When Each One Wins

Networking11 min

The OSI Model Will Finally Make Sense After This

Git11 min

The 7 Levels of Git

Networking10 min

Every Network Troubleshooting Command You Need

Cybersecurity10 min

Hashing vs Encryption in Cybersecurity: When Each One Wins

Networking11 min

Why the Internet Ran Out of IP Addresses

Linux11 min

Linux Package Managers: apt vs Snap vs Flatpak

Python9 min

Why Python Is Slow (and When It Doesn't Matter)

Networking11 min

TCP vs UDP: When Each One Wins

Linux8 min

Stop Running Linux as Root (Do This Instead)

Linux11 min

Linux Users and Groups Will Finally Make Sense

Cybersecurity9 min

Why Phishing Beats Good Security (and How to Spot It)

Docker12 min

The 7 Levels of Docker

AI agents & MCP12 min

AI Agents Will Finally Make Sense After This

Git8 min

Stop Using git push --force (Use This Instead)

AI agent skills10 min

Agent Skills vs MCP: When Each One Wins

Networking12 min

Networking Will Finally Make Sense After This

Linux12 min

Every Linux Command You Actually Need

Cybersecurity12 min

The 7 Levels of Cybersecurity

System design12 min

Every System Design Concept You Need, in One Diagram

PostgreSQL10 min

Why PostgreSQL Ignores Your Index