SRE Best Practices: SLOs, Error Budgets & Chaos Engineering
Site Reliability Engineering (SRE) principles: calculating SLI, SLO, SLA, managing Error Budgets, and injecting deliberate failures with Chaos Engineering.
SRE Best Practices: SLOs, Error Budgets & Chaos Engineering
Section titled “SRE Best Practices: SLOs, Error Budgets & Chaos Engineering”Site Reliability Engineering (SRE) balances the need for product velocity with the requirement for system reliability. Instead of aiming for an unrealistic “100% uptime,” SRE embraces failure and defines measurable boundaries.
flowchart LR SLA["<b>SLA</b><br/>Business Agreement<br/>(Contractual Penalties)"] --> SLO["<b>SLO</b><br/>Internal Target<br/>(e.g. 99.9% Uptime)"] SLO --> SLI["<b>SLI</b><br/>Direct Measurement<br/>(Prometheus Metric)"] SLO --> EB["<b>Error Budget</b><br/>0.1% Allowed Failure<br/>(Fuel for Innovation)"]
style SLA fill:#fff1f2,stroke:#f43f5e,stroke-width:2px,rx:8px,ry:8px style SLO fill:#eff6ff,stroke:#2563eb,stroke-width:2px,rx:8px,ry:8px style SLI fill:#f0fdf4,stroke:#16a34a,stroke-width:2px,rx:8px,ry:8px style EB fill:#fef3c7,stroke:#f59e0b,stroke-width:2px,rx:8px,ry:8pxSLI vs SLO vs SLA Defined
Section titled “SLI vs SLO vs SLA Defined”| Metric | What It Represents | Example |
|---|---|---|
| SLI (Service Level Indicator) | A quantifiable metric measuring real-time performance. | Proportion of successful HTTP requests (2xx/3xx) over total requests. |
| SLO (Service Level Objective) | The internal target agreed upon by engineering and product teams. | “99.9% of requests over any 30-day window must succeed in < 200ms.” |
| SLA (Service Level Agreement) | A legal or commercial contract with customers specifying financial refunds. | “If availability drops below 99.5%, customer receives a 10% billing credit.” |
The Power of Error Budgets
Section titled “The Power of Error Budgets”The Error Budget is 100% - SLO.
For a 99.9% SLO (“Three Nines”), your allowed downtime across a 30-day period is: $$\text{Allowed Downtime} = 30 \text{ days} \times 24 \times 60 \times (1 - 0.999) = 43.2 \text{ minutes}$$
- When Error Budget is Green (> 0%): Developers have full freedom to ship experimental features, perform migrations, and push aggressive releases.
- When Error Budget is Exhausted (0%): Feature deployments are frozen immediately. All engineering focus pivots to technical debt, reliability, bug fixes, and testing.
Chaos Engineering: Designing for Inevitable Failure
Section titled “Chaos Engineering: Designing for Inevitable Failure”Chaos Engineering is the discipline of experimenting on a distributed software system to build confidence in its capability to withstand turbulent production conditions.
flowchart TD Hypothesis["1. Formulate Hypothesis<br/>'If AZ-1 fails, traffic automatically reroutes to AZ-2 in < 15s'"] Inject["2. Inject Real Fault<br/>(Chaos Mesh / Litmus / AWS FIS)"] Observe["3. Observe Impact<br/>(Prometheus SLIs & Grafana)"] Analyze["4. Analyze Difference & Fix<br/>(Auto Scaling, Health Check timeouts)"]
Hypothesis --> Inject --> Observe --> Analyze --> Hypothesis
style Hypothesis fill:#eff6ff,stroke:#3b82f6 style Inject fill:#fef2f2,stroke:#ef4444 style Observe fill:#f0fdf4,stroke:#22c55e style Analyze fill:#faf5ff,stroke:#a855f7Common Chaos Scenarios in Cloud & Kubernetes
Section titled “Common Chaos Scenarios in Cloud & Kubernetes”- Pod Termination: Randomly killing random pods (
pod-kill) to verify StatefulSet and Deployment replica resilience. - Network Latency / Packet Loss: Injecting 200ms artificial latency between services to test circuit breakers and timeout policies.
- Zone Outage Simulation: Blocking network routing to an entire Availability Zone to ensure load balancers cleanly shed uncontactable instances.