How to delete failed, error and CrashLoopBackOff pods in Kubernetes (k8s)
After a series of unexpected power failures and hardware issues, I had a bunch of failed pods in my Kubernetes cluster. How do I delete all failed pods in Kubernetes?
DO THIS WITH CAUTION - This will delete all failed/error pods in your Kubernetes cluster.
Here is a selection of Error and Crash Loop Back Off pods in my Kubernetes cluster.
```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running 6 (9m35s ago) 130d
hello-world-6cb7cd95b5-z7qhr 1/1 Running 7 (9m35s ago) 142d
kube-prometheus-stack-grafana-7664d8545c-k4bmx 0/3 Completed 0 96d
kube-prometheus-stack-grafana-7664d8545c-k4tgq 3/3 Running 6 (9m36s ago) 96d
kube-prometheus-stack-grafana-7664d8545c-n7zdj 0/3 Completed 3 (127d ago) 130d
kube-prometheus-stack-grafana-7664d8545c-r7xdj 0/3 Completed 0 99d
kube-prometheus-stack-kube-state-metrics-69d546c6c-j9srj 1/1 Running 3 (9m36s ago) 130d
kube-prometheus-stack-operator-74cf4c448b-8lq44 1/1 Running 3 (9m36s ago) 130d
kube-prometheus-stack-prometheus-node-exporter-27m78 0/1 CrashLoopBackOff 26525 (2m29s ago) 94d
kube-prometheus-stack-prometheus-node-exporter-pm6dk 0/1 CrashLoopBackOff 966 (2m15s ago) 3d9h
kube-prometheus-stack-prometheus-node-exporter-xpd89 0/1 CrashLoopBackOff 309 (2m29s ago) 25h
metrics-server-699ffcdc79-t5lsw 1/1 Running 6 (9m36s ago) 135d
my-foldingathome-9f7fdb678-bjr6j 0/1 Error 1 128d
my-foldingathome-9f7fdb678-qsc57 1/1 Running 1 (20h ago) 95d
my-squid-76b7d89496-7mzn9 1/1 Running 3 (9m36s ago) 46d
nfs-subdir-external-provisioner-66b5dff8cf-xlzgn 1/1 Running 9 (20h ago) 139d
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 4 (9m36s ago) 3d9h
rabbit-rabbitmq-0 1/1 Running 0 16m
wiki-js-58cb544c76-wr2zc 0/1 CrashLoopBackOff 244 (2m7s ago) 139d
wiki-js-postgresql-0 1/1 Running 6 (20h ago) 139d
List all failed pods
Run the following command to list all failed pods in your Kubernetes cluster.
$ kubectl get pods -A | grep -v Running
Example output:
[~] $ kubectl get pods -A | grep -v Running
NAMESPACE NAME READY STATUS RESTARTS AGE
default kube-prometheus-stack-grafana-7664d8545c-k4bmx 0/3 Completed 0 96d
default kube-prometheus-stack-grafana-7664d8545c-n7zdj 0/3 Completed 3 (127d ago) 130d
default kube-prometheus-stack-grafana-7664d8545c-r7xdj 0/3 Completed 0 99d
default kube-prometheus-stack-prometheus-node-exporter-27m78 0/1 CrashLoopBackOff 26525 (3m30s ago) 94d
default kube-prometheus-stack-prometheus-node-exporter-pm6dk 0/1 CrashLoopBackOff 966 (3m16s ago) 3d9h
default kube-prometheus-stack-prometheus-node-exporter-xpd89 0/1 CrashLoopBackOff 309 (3m30s ago) 25h
default my-foldingathome-9f7fdb678-bjr6j 0/1 Error 1 128d
Double check the list
To get list of pods that are in Error or CrashLoopBackOff state, run the following command.
kubectl get pods --all-namespaces | grep -v Running |awk '{print $2 " --namespace=" $1}' | grep -v ^NAME
Example output:
[~] $ kubectl get pods --all-namespaces | grep -v Running |awk '{print $2 " --namespace=" $1}' | grep -v ^NAME
kube-prometheus-stack-grafana-7664d8545c-k4bmx --namespace=default
kube-prometheus-stack-grafana-7664d8545c-n7zdj --namespace=default
kube-prometheus-stack-grafana-7664d8545c-r7xdj --namespace=default
kube-prometheus-stack-prometheus-node-exporter-27m78 --namespace=default
kube-prometheus-stack-prometheus-node-exporter-pm6dk --namespace=default
kube-prometheus-stack-prometheus-node-exporter-xpd89 --namespace=default
Delete all failed pods
Once you are happy with the list of failed pods, you can delete them using the following command.
kubectl get pods --all-namespaces | grep -v Running |awk '{print $2 " --namespace=" $1}' | grep -v ^NAME | xargs kubectl delete pod
Example output:
[~] $ kubectl get pods --all-namespaces | grep -v Running |awk '{print $2 " --namespace=" $1}' | grep -v ^NAME | xargs kubectl delete pod
pod "kube-prometheus-stack-grafana-7664d8545c-k4bmx" deleted
pod "kube-prometheus-stack-grafana-7664d8545c-n7zdj" deleted
pod "kube-prometheus-stack-grafana-7664d8545c-r7xdj" deleted
pod "kube-prometheus-stack-prometheus-node-exporter-27m78" deleted
pod "kube-prometheus-stack-prometheus-node-exporter-pm6dk" deleted
pod "kube-prometheus-stack-prometheus-node-exporter-xpd89" deleted