Kubernetes Troubleshooting Guide

by Bala

Effective troubleshooting and debugging practices for diagnosing common Kubernetes cluster and workload issues

Kubernetes Troubleshooting Guide

Diagnosing issues in a Kubernetes cluster means narrowing down whether the problem is in the pod, the node, the network, or the control plane. This guide walks through the commands and checks that cover most day-to-day incidents.

Prerequisites

  • kubectl configured against the target cluster
  • Sufficient RBAC permissions to read pods, events, and logs across namespaces
  • jq installed for filtering JSON output (optional but useful)

Pods Stuck in Pending

A pending pod hasn’t been scheduled onto a node yet. Start with the scheduler’s own explanation:

kubectl describe pod <pod-name> -n <namespace>

Look at the Events section at the bottom of the output. Common causes:

# Insufficient CPU/memory on any node
kubectl describe nodes | grep -A5 "Allocated resources"

# No node matches the pod's nodeSelector/affinity rules
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeSelector}'

# PersistentVolumeClaim not bound
kubectl get pvc -n <namespace>

Pods Stuck in CrashLoopBackOff

The container is starting and exiting repeatedly. Get the logs from the previous attempt, not the current one:

kubectl logs <pod-name> -n <namespace> --previous
kubectl describe pod <pod-name> -n <namespace>

Frequent root causes:

  • The application crashes on startup (missing config, failed dependency connection) — check the log output above.
  • A liveness probe is failing before the app finishes starting — increase initialDelaySeconds or use a startupProbe.
  • OOMKilled — check kubectl describe pod for Reason: OOMKilled and raise memory limits.
# Confirm whether the last restart was an OOM kill
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Pods Stuck in ImagePullBackOff

kubectl describe pod <pod-name> -n <namespace>

Check for:

  • A typo in the image name/tag
  • A private registry without an imagePullSecrets entry on the pod or service account
  • Registry rate limiting (common with unauthenticated Docker Hub pulls)
# Verify the pull secret is attached
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

Service Not Reachable

Work outward from the pod to the service to the client:

# 1. Confirm the pod itself is healthy and listening on the expected port
kubectl exec -it <pod-name> -n <namespace> -- curl -sv localhost:<port>/health

# 2. Confirm the Service selects the pod
kubectl get endpoints <service-name> -n <namespace>

# 3. If the endpoint list is empty, the label selector doesn't match
kubectl get pods -n <namespace> --show-labels
kubectl get svc <service-name> -n <namespace> -o jsonpath='{.spec.selector}'

# 4. Test resolution and connectivity from inside the cluster
kubectl run debug --rm -it --image=nicolaka/netshoot -- curl -sv http://<service-name>.<namespace>.svc.cluster.local

If step 4 fails but step 1 succeeds, check for a NetworkPolicy blocking traffic:

kubectl get networkpolicies -n <namespace>
kubectl describe networkpolicy <policy-name> -n <namespace>

Node Issues

# Node conditions (MemoryPressure, DiskPressure, PIDPressure, Ready)
kubectl describe node <node-name>

# Pods evicted due to node pressure
kubectl get events --field-selector reason=Evicted -A

# kubelet logs (requires SSH access to the node)
journalctl -u kubelet -n 200 --no-pager

Cluster-Wide Triage Commands

# All non-running pods across the cluster
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

# Recent warning events, most recent last
kubectl get events -A --sort-by='.lastTimestamp' | grep Warning

# Resource pressure per node
kubectl top nodes
kubectl top pods -A --sort-by=memory

Common Pitfalls

  1. Reading kubectl logs without --previous on a container that already restarted
  2. Debugging a Service before checking whether it has any endpoints at all
  3. Assuming a Pending pod is a networking issue when it’s actually unschedulable
  4. Missing readinessProbe failures because the pod shows Running but 0/1 Ready

Additional Resources