Production Observability & Monitoring Foundations
Master modern cloud observability: the Three Pillars (Metrics, Logs, Traces), Prometheus scraping architectures, Grafana dashboarding, and alerting hygiene.
Production Observability & Monitoring Foundations
Section titled “Production Observability & Monitoring Foundations”Monitoring tells you when a system is broken. Observability lets you understand why it broke—even for complex, novel failure modes across distributed microservices.
flowchart TD subgraph Pillars["The Three Pillars of Observability"] M["📊 Metrics (Prometheus)<br/>'Is something wrong, and where?'<br/>Counters • Gauges • Histograms"] L["📜 Logs (Loki / Fluentbit)<br/>'What specifically happened?'<br/>Structured JSON • Trace Context"] T["🔍 Traces (OpenTelemetry)<br/>'Where did the request spend time?'<br/>Spans • Distributed Latency"] end
style Pillars fill:#eff6ff,stroke:#2563eb,stroke-width:2px,rx:10px,ry:10pxMetrics: Time-Series with Prometheus & PromQL
Section titled “Metrics: Time-Series with Prometheus & PromQL”Prometheus uses a pull-based scraping model where a central server queries /metrics HTTP endpoints on services, nodes, and containers.
Core Prometheus Metric Types
Section titled “Core Prometheus Metric Types”| Type | Definition | Production Example |
|---|---|---|
| Counter | Cumulative value that only increases or resets to zero on restart. | http_requests_total, packet_drop_count |
| Gauge | Value that goes up and down arbitrarily. | node_memory_utilization_bytes, cpu_usage_percent |
| Histogram | Samples observations (usually duration/size) into configurable buckets. | http_request_duration_seconds_bucket (calculating p95, p99 latency) |
| Summary | Similar to Histogram, calculates streaming quantiles on the client side. | High-frequency RPC latencies |
Golden PromQL Queries for SREs
Section titled “Golden PromQL Queries for SREs”# 1. Real-time request rate per second over the last 5 minutessum(rate(http_requests_total[5m])) by (status)
# 2. 99th percentile (p99) API response latencyhistogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# 3. Memory pressure alert: percentage of free memory on Linux nodes(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10Distributed Tracing: OpenTelemetry & Spans
Section titled “Distributed Tracing: OpenTelemetry & Spans”In microservice architectures, a single user click might travel through an API Gateway, Auth Service, Billing Service, and Database. Distributed tracing propagates a Trace ID across HTTP headers (traceparent), tracking execution duration at each step:
[User Request] ──(Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736)── │ ├─► [API Gateway Ingress] (Span 1: 15ms) ├─► [Authentication Pod] (Span 2: 45ms) ├─► [Billing Database] (Span 3: 210ms - BOTTLENECK!) └─► [Notification Worker] (Span 4: 10ms)The 4 Golden Signals of SRE (Google SRE Framework)
Section titled “The 4 Golden Signals of SRE (Google SRE Framework)”Every production dashboard and alert must be structured around Google’s Four Golden Signals:
- Latency: The time it takes to service a request (differentiate successful request latency from error latency).
- Traffic: A measure of how much demand is placed on your system (HTTP req/sec, network I/O).
- Errors: The rate of requests that fail (HTTP 500 errors, broken database connections).
- Saturation: How “full” your service is (memory saturation, CPU thread starvation, connection pool exhaustion).