Skip to content
About Me
----

SRE On-Call Emergency Runbook & Troubleshooting Cheat Sheet

Architect Key Takeaways

High-pressure, production-tested cheat sheet for Site Reliability Engineers and DevOps on-call responders. Rapid triage commands for Kubernetes, Linux kernel, networking, etcd, and Prometheus.

SRE On-Call Emergency Runbook & Troubleshooting Cheat Sheet

Section titled β€œSRE On-Call Emergency Runbook & Troubleshooting Cheat Sheet”

When a SEV-1 Critical Outage strikes at 3:00 AM, there is no time to browse lengthy documentation. This runbook provides instant, copy-pasteable triage commands, diagnostics, and recovery recipes tested on production-scale clusters.


1. The Emergency Triage Golden Sequence (First 5 Minutes)

Section titled β€œ1. The Emergency Triage Golden Sequence (First 5 Minutes)”
[ PAGERDUTY ALERT FIRES ]
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Step 1: Establish Incident Bridge & Silence β”‚
β”‚ β€’ Join Incident Zoom / Slack war room β”‚
β”‚ β€’ Post: "Incident Commander (IC) established" β”‚
β”‚ β€’ Freeze deployments: kubectl scale / git lock β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Step 2: High-Level Blast Radius Assessment β”‚
β”‚ β€’ Identify affected regions & microservices β”‚
β”‚ β€’ Check Cloud Provider Status dashboard β”‚
β”‚ β€’ Run cluster vital signs triage script β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Step 3: Mitigate Customer Impact (Rollback/Drain)β”‚
β”‚ β€’ Mitigation BEFORE Root Cause! β”‚
β”‚ β€’ Rollback recent git commits or canary traffic β”‚
β”‚ β€’ Scale surviving healthy replicas β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Terminal window
# 1. Find all non-running, crashing, or evicting pods across the entire cluster
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
# 2. Sort pods by restart count to identify cyclic crashloops
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -n 25
# 3. View the logs of the PREVIOUS instance of a crashed container (post-mortem)
kubectl logs <pod-name> -n <namespace> --previous --tail=100
# 4. Grep recent cluster warning and failure events sorted by timestamp
kubectl get events -A --sort-by='.metadata.creationTimestamp' | grep -iE 'failed|unhealthy|killing|evicted|backoff' | tail -n 30
Terminal window
# Check node conditions for MemoryPressure, DiskPressure, or PIDPressure
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type,REASON:.status.conditions[-1].reason
# Check live CPU and Memory resource consumption per node
kubectl top nodes --sort-by=memory
# Check live top memory-consuming pods in production
kubectl top pods -n production --sort-by=memory --no-headers | head -n 15

3. Ephemeral Debugging in Zero-Shell / Distroless Containers

Section titled β€œ3. Ephemeral Debugging in Zero-Shell / Distroless Containers”

When a distroless container crashes and kubectl exec is disabled or missing /bin/sh:

Terminal window
# Inject an ephemeral debugging container with full network and curl tools into the live target pod
kubectl debug -it <pod-name> -n <namespace> \
--image=nicolaka/netshoot:latest \
--target=<target-container-name> \
-- bash

Inside the netshoot container, run immediate connectivity and DNS diagnostics:

Terminal window
# Test CoreDNS resolution
nslookup kubernetes.default.svc.cluster.local
dig +short payment-service.finance.svc.cluster.local
# Test Layer 7 HTTP latency and TLS certificate expiration
curl -Iv https://internal-api.service.local/healthz

When investigating high CPU load, unkillable zombie processes, or frozen disks on worker nodes:

Terminal window
# Check CPU steal time, load average, and context switching (look at %steal for noisy cloud neighbors)
vmstat 1 5
# Identify top processes consuming Linux File Descriptors (FD leak)
lsof | awk '{print $1}' | sort | uniq -c | sort -rn | head -n 15
# Check dmesg for Linux OOM (Out-of-Memory) Killer invocations
dmesg -T | grep -iE 'oom[-_]killer|killed process' | tail -n 20
Terminal window
# Count active socket states (diagnose SYN flood or TIME_WAIT socket exhaustion)
ss -ant | awk '{print $1}' | sort | uniq -c
# Check dropped packets and network interface errors
ip -s link show eth0

Terminal window
# Perform a controlled zero-downtime rolling restart of all pods in a deployment
kubectl rollout restart deployment/payment-api -n production
# Monitor status until 100% healthy
kubectl rollout status deployment/payment-api -n production --timeout=120s

Recipe 2: Emergency Immediate Rollback to Previous Version

Section titled β€œRecipe 2: Emergency Immediate Rollback to Previous Version”
Terminal window
# View deployment revision history
kubectl rollout history deployment/payment-api -n production
# Undo immediate bad deployment to previous known healthy revision
kubectl rollout undo deployment/payment-api -n production
Terminal window
# 1. Cordon the node (prevent new pods from scheduling)
kubectl cordon ip-10-0-2-45.ec2.internal
# 2. Drain workloads respecting PodDisruptionBudgets (PDBs)
kubectl drain ip-10-0-2-45.ec2.internal \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=30 \
--force

Recipe 4: Fixing CoreDNS Crashloop & Cluster DNS Outage

Section titled β€œRecipe 4: Fixing CoreDNS Crashloop & Cluster DNS Outage”
Terminal window
# Scale CoreDNS to handle emergency traffic storm
kubectl scale deployment/coredns -n kube-system --replicas=5
# Check if CoreDNS has upstream loop detection issues
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

  1. Mitigate First, Investigate Second: If rolling back the last commit or spinning up 10 extra replicas stops customer downtime, execute the mitigation immediately before diagnosing the root cause.
  2. Never Make Undocumented Console Changes: If you perform an emergency manual fix, document the command in the incident channel immediately so GitOps controllers (ArgoCD) do not overwrite it mid-incident.
  3. Preserve Volatile Memory: Before killing a hung Java or Go pod, capture a thread dump or heap dump if time permits (kill -3 <pid> or jcmd <pid> Thread.print).