Skip to content
About Me
----

SRE Best Practices: SLOs, Error Budgets & Chaos Engineering

Architect Key Takeaways

Site Reliability Engineering (SRE) principles: calculating SLI, SLO, SLA, managing Error Budgets, and injecting deliberate failures with Chaos Engineering.

SRE Best Practices: SLOs, Error Budgets & Chaos Engineering

Section titled “SRE Best Practices: SLOs, Error Budgets & Chaos Engineering”

Site Reliability Engineering (SRE) balances the need for product velocity with the requirement for system reliability. Instead of aiming for an unrealistic “100% uptime,” SRE embraces failure and defines measurable boundaries.

flowchart LR
SLA["<b>SLA</b><br/>Business Agreement<br/>(Contractual Penalties)"] --> SLO["<b>SLO</b><br/>Internal Target<br/>(e.g. 99.9% Uptime)"]
SLO --> SLI["<b>SLI</b><br/>Direct Measurement<br/>(Prometheus Metric)"]
SLO --> EB["<b>Error Budget</b><br/>0.1% Allowed Failure<br/>(Fuel for Innovation)"]
style SLA fill:#fff1f2,stroke:#f43f5e,stroke-width:2px,rx:8px,ry:8px
style SLO fill:#eff6ff,stroke:#2563eb,stroke-width:2px,rx:8px,ry:8px
style SLI fill:#f0fdf4,stroke:#16a34a,stroke-width:2px,rx:8px,ry:8px
style EB fill:#fef3c7,stroke:#f59e0b,stroke-width:2px,rx:8px,ry:8px

Metric What It Represents Example
SLI (Service Level Indicator) A quantifiable metric measuring real-time performance. Proportion of successful HTTP requests (2xx/3xx) over total requests.
SLO (Service Level Objective) The internal target agreed upon by engineering and product teams. “99.9% of requests over any 30-day window must succeed in < 200ms.”
SLA (Service Level Agreement) A legal or commercial contract with customers specifying financial refunds. “If availability drops below 99.5%, customer receives a 10% billing credit.”

The Error Budget is 100% - SLO.

For a 99.9% SLO (“Three Nines”), your allowed downtime across a 30-day period is: $$\text{Allowed Downtime} = 30 \text{ days} \times 24 \times 60 \times (1 - 0.999) = 43.2 \text{ minutes}$$

  • When Error Budget is Green (> 0%): Developers have full freedom to ship experimental features, perform migrations, and push aggressive releases.
  • When Error Budget is Exhausted (0%): Feature deployments are frozen immediately. All engineering focus pivots to technical debt, reliability, bug fixes, and testing.

Chaos Engineering: Designing for Inevitable Failure

Section titled “Chaos Engineering: Designing for Inevitable Failure”

Chaos Engineering is the discipline of experimenting on a distributed software system to build confidence in its capability to withstand turbulent production conditions.

flowchart TD
Hypothesis["1. Formulate Hypothesis<br/>'If AZ-1 fails, traffic automatically reroutes to AZ-2 in < 15s'"]
Inject["2. Inject Real Fault<br/>(Chaos Mesh / Litmus / AWS FIS)"]
Observe["3. Observe Impact<br/>(Prometheus SLIs & Grafana)"]
Analyze["4. Analyze Difference & Fix<br/>(Auto Scaling, Health Check timeouts)"]
Hypothesis --> Inject --> Observe --> Analyze --> Hypothesis
style Hypothesis fill:#eff6ff,stroke:#3b82f6
style Inject fill:#fef2f2,stroke:#ef4444
style Observe fill:#f0fdf4,stroke:#22c55e
style Analyze fill:#faf5ff,stroke:#a855f7

Common Chaos Scenarios in Cloud & Kubernetes

Section titled “Common Chaos Scenarios in Cloud & Kubernetes”
  1. Pod Termination: Randomly killing random pods (pod-kill) to verify StatefulSet and Deployment replica resilience.
  2. Network Latency / Packet Loss: Injecting 200ms artificial latency between services to test circuit breakers and timeout policies.
  3. Zone Outage Simulation: Blocking network routing to an entire Availability Zone to ensure load balancers cleanly shed uncontactable instances.