SRE On-Call Emergency Runbook & Troubleshooting Cheat Sheet
Architect Key Takeaways
High-pressure, production-tested cheat sheet for Site Reliability Engineers and DevOps on-call responders. Rapid triage commands for Kubernetes, Linux kernel, networking, etcd, and Prometheus.
SRE On-Call Emergency Runbook & Troubleshooting Cheat Sheet
Section titled βSRE On-Call Emergency Runbook & Troubleshooting Cheat SheetβWhen a SEV-1 Critical Outage strikes at 3:00 AM, there is no time to browse lengthy documentation. This runbook provides instant, copy-pasteable triage commands, diagnostics, and recovery recipes tested on production-scale clusters.
1. The Emergency Triage Golden Sequence (First 5 Minutes)
Section titled β1. The Emergency Triage Golden Sequence (First 5 Minutes)β [ PAGERDUTY ALERT FIRES ] β βΌ ββββββββββββββββββββββββββββββββββββββββββββββββββββ β Step 1: Establish Incident Bridge & Silence β β β’ Join Incident Zoom / Slack war room β β β’ Post: "Incident Commander (IC) established" β β β’ Freeze deployments: kubectl scale / git lock β ββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββ β βΌ ββββββββββββββββββββββββββββββββββββββββββββββββββββ β Step 2: High-Level Blast Radius Assessment β β β’ Identify affected regions & microservices β β β’ Check Cloud Provider Status dashboard β β β’ Run cluster vital signs triage script β ββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββ β βΌ ββββββββββββββββββββββββββββββββββββββββββββββββββββ β Step 3: Mitigate Customer Impact (Rollback/Drain)β β β’ Mitigation BEFORE Root Cause! β β β’ Rollback recent git commits or canary traffic β β β’ Scale surviving healthy replicas β ββββββββββββββββββββββββββββββββββββββββββββββββββββ2. Kubernetes Emergency Diagnostic Commands
Section titled β2. Kubernetes Emergency Diagnostic Commandsβ2.1 Pods & Workloads Triage
Section titled β2.1 Pods & Workloads Triageβ# 1. Find all non-running, crashing, or evicting pods across the entire clusterkubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
# 2. Sort pods by restart count to identify cyclic crashloopskubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -n 25
# 3. View the logs of the PREVIOUS instance of a crashed container (post-mortem)kubectl logs <pod-name> -n <namespace> --previous --tail=100
# 4. Grep recent cluster warning and failure events sorted by timestampkubectl get events -A --sort-by='.metadata.creationTimestamp' | grep -iE 'failed|unhealthy|killing|evicted|backoff' | tail -n 302.2 Node Pressure & Resource Starvation
Section titled β2.2 Node Pressure & Resource Starvationβ# Check node conditions for MemoryPressure, DiskPressure, or PIDPressurekubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type,REASON:.status.conditions[-1].reason
# Check live CPU and Memory resource consumption per nodekubectl top nodes --sort-by=memory
# Check live top memory-consuming pods in productionkubectl top pods -n production --sort-by=memory --no-headers | head -n 153. Ephemeral Debugging in Zero-Shell / Distroless Containers
Section titled β3. Ephemeral Debugging in Zero-Shell / Distroless ContainersβWhen a distroless container crashes and kubectl exec is disabled or missing /bin/sh:
# Inject an ephemeral debugging container with full network and curl tools into the live target podkubectl debug -it <pod-name> -n <namespace> \ --image=nicolaka/netshoot:latest \ --target=<target-container-name> \ -- bashInside the netshoot container, run immediate connectivity and DNS diagnostics:
# Test CoreDNS resolutionnslookup kubernetes.default.svc.cluster.localdig +short payment-service.finance.svc.cluster.local
# Test Layer 7 HTTP latency and TLS certificate expirationcurl -Iv https://internal-api.service.local/healthz4. Linux Kernel & Host Level Diagnostics
Section titled β4. Linux Kernel & Host Level DiagnosticsβWhen investigating high CPU load, unkillable zombie processes, or frozen disks on worker nodes:
4.1 CPU & Process Bottlenecks
Section titled β4.1 CPU & Process Bottlenecksβ# Check CPU steal time, load average, and context switching (look at %steal for noisy cloud neighbors)vmstat 1 5
# Identify top processes consuming Linux File Descriptors (FD leak)lsof | awk '{print $1}' | sort | uniq -c | sort -rn | head -n 15
# Check dmesg for Linux OOM (Out-of-Memory) Killer invocationsdmesg -T | grep -iE 'oom[-_]killer|killed process' | tail -n 204.2 Network & Socket Saturation
Section titled β4.2 Network & Socket Saturationβ# Count active socket states (diagnose SYN flood or TIME_WAIT socket exhaustion)ss -ant | awk '{print $1}' | sort | uniq -c
# Check dropped packets and network interface errorsip -s link show eth05. Emergency Recovery Recipes
Section titled β5. Emergency Recovery RecipesβRecipe 1: Rolling Restart of a Stuck Microservice
Section titled βRecipe 1: Rolling Restart of a Stuck Microserviceβ# Perform a controlled zero-downtime rolling restart of all pods in a deploymentkubectl rollout restart deployment/payment-api -n production
# Monitor status until 100% healthykubectl rollout status deployment/payment-api -n production --timeout=120sRecipe 2: Emergency Immediate Rollback to Previous Version
Section titled βRecipe 2: Emergency Immediate Rollback to Previous Versionβ# View deployment revision historykubectl rollout history deployment/payment-api -n production
# Undo immediate bad deployment to previous known healthy revisionkubectl rollout undo deployment/payment-api -n productionRecipe 3: Safely Cordon & Drain a Malfunctioning Node
Section titled βRecipe 3: Safely Cordon & Drain a Malfunctioning Nodeβ# 1. Cordon the node (prevent new pods from scheduling)kubectl cordon ip-10-0-2-45.ec2.internal
# 2. Drain workloads respecting PodDisruptionBudgets (PDBs)kubectl drain ip-10-0-2-45.ec2.internal \ --ignore-daemonsets \ --delete-emptydir-data \ --grace-period=30 \ --forceRecipe 4: Fixing CoreDNS Crashloop & Cluster DNS Outage
Section titled βRecipe 4: Fixing CoreDNS Crashloop & Cluster DNS Outageβ# Scale CoreDNS to handle emergency traffic stormkubectl scale deployment/coredns -n kube-system --replicas=5
# Check if CoreDNS has upstream loop detection issueskubectl logs -n kube-system -l k8s-app=kube-dns --tail=506. On-Call Golden Rules
Section titled β6. On-Call Golden Rulesβ- Mitigate First, Investigate Second: If rolling back the last commit or spinning up 10 extra replicas stops customer downtime, execute the mitigation immediately before diagnosing the root cause.
- Never Make Undocumented Console Changes: If you perform an emergency manual fix, document the command in the incident channel immediately so GitOps controllers (ArgoCD) do not overwrite it mid-incident.
- Preserve Volatile Memory: Before killing a hung Java or Go pod, capture a thread dump or heap dump if time permits (
kill -3 <pid>orjcmd <pid> Thread.print).