Kubernetes troubleshooting checklist
Troubleshooting is 30% of the CKA. The fastest way through is the same order every time: read what the cluster already tells you, then go one level lower.
Last reviewed . Paths and labels are those of a kubeadm cluster.
A node is NotReady
Usually the kubelet: stopped, misconfigured or unable to reach the API server.
- See which node and since when
kubectl get nodes -o wide - Read the node's conditions and recent events
kubectl describe node node01 - On the node: is the kubelet running?
systemctl status kubelet - Why it stopped or keeps restarting: read the last lines first
journalctl -u kubelet --no-pager | tail -30 - Check the files it names: config, kubeconfig, certificates
cat /var/lib/kubelet/config.yaml - Is the container runtime up?
systemctl status containerd - Fix, then restart and confirm
systemctl daemon-reload && systemctl restart kubelet
kubectl can't reach the cluster
On kubeadm, the API server and etcd are static Pods. When they're down, kubectl can't tell you why, so go one level lower.
- Is it the cluster or your kubeconfig? Check server and context
kubectl config view --minify - Which control-plane containers are running or exiting?
crictl ps -a - Why a container exits
crictl logs <container-id> - If no container exists at all, the manifest may not parse
journalctl -u kubelet | grep -i error | tail - Compare flags, paths and ports with a known-good value
grep -n -- '--etcd\|--advertise\|--client-ca' /etc/kubernetes/manifests/kube-apiserver.yaml - Wait for the kubelet to pick up your fix, then confirm
kubectl get pods -n kube-system
A Pod is Pending
The scheduler can't place it, or something it needs doesn't exist yet.
- Read the scheduler's reason in the events
kubectl describe pod <pod> -n <ns> - Not enough CPU or memory? Compare requests with what's free
kubectl describe node node01 | grep -A8 Allocated - Taints the Pod doesn't tolerate
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints - Node selectors or affinity that match no node
kubectl get nodes --show-labels - An unbound PersistentVolumeClaim
kubectl get pvc -n <ns> - Is the scheduler itself running?
kubectl get pods -n kube-system -l component=kube-scheduler
A Pod crashes or won't start
Look at the container's last words before changing anything.
- The status tells you the category
kubectl get pod <pod> -n <ns> - ImagePullBackOff: check the image name, tag and pull secret
kubectl describe pod <pod> -n <ns> - CrashLoopBackOff: read the log of the previous attempt
kubectl logs <pod> -n <ns> --previous - OOMKilled: compare the memory limit with real use
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].lastState}' - Probes failing: check path, port and timing against the app
kubectl get pod <pod> -n <ns> -o yaml | grep -A6 Probe - Missing ConfigMap, Secret or volume: events say which
kubectl get events -n <ns> --sort-by=.lastTimestamp
A Service doesn't answer
Work from the Pods outwards: Pods, selector, endpoints, ports, policy.
- Are the backing Pods Running and Ready?
kubectl get pods -n <ns> -o wide --show-labels - Does the Service selector match their labels?
kubectl get svc <svc> -n <ns> -o wide - Does it have endpoints? None means selector or readiness
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<svc> - Does targetPort match the port the container listens on?
kubectl get svc <svc> -n <ns> -o yaml - Is a NetworkPolicy blocking it?
kubectl get netpol -n <ns> - Test from inside the cluster
kubectl run t --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- -T2 http://<svc>.<ns>
Names don't resolve
Usually CoreDNS isn't running, its Service has no endpoints, or its Corefile is wrong.
- Are the CoreDNS Pods running?
kubectl get pods -n kube-system -l k8s-app=kube-dns - Does the kube-dns Service have endpoints?
kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns - What does CoreDNS log?
kubectl logs -n kube-system -l k8s-app=kube-dns - Is the Corefile sane?
kubectl get cm coredns -n kube-system -o yaml - Resolve from a Pod
kubectl run t --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
Practise on broken clusters
Each of these failures has a lab: we break a real cluster in exactly this way and check that you fixed it.