Skip to content
About Me
----

Production Observability & Monitoring Foundations

Architect Key Takeaways

Master modern cloud observability: the Three Pillars (Metrics, Logs, Traces), Prometheus scraping architectures, Grafana dashboarding, and alerting hygiene.

Production Observability & Monitoring Foundations

Section titled “Production Observability & Monitoring Foundations”

Monitoring tells you when a system is broken. Observability lets you understand why it broke—even for complex, novel failure modes across distributed microservices.

flowchart TD
subgraph Pillars["The Three Pillars of Observability"]
M["📊 Metrics (Prometheus)<br/>'Is something wrong, and where?'<br/>Counters • Gauges • Histograms"]
L["📜 Logs (Loki / Fluentbit)<br/>'What specifically happened?'<br/>Structured JSON • Trace Context"]
T["🔍 Traces (OpenTelemetry)<br/>'Where did the request spend time?'<br/>Spans • Distributed Latency"]
end
style Pillars fill:#eff6ff,stroke:#2563eb,stroke-width:2px,rx:10px,ry:10px

Metrics: Time-Series with Prometheus & PromQL

Section titled “Metrics: Time-Series with Prometheus & PromQL”

Prometheus uses a pull-based scraping model where a central server queries /metrics HTTP endpoints on services, nodes, and containers.

Type Definition Production Example
Counter Cumulative value that only increases or resets to zero on restart. http_requests_total, packet_drop_count
Gauge Value that goes up and down arbitrarily. node_memory_utilization_bytes, cpu_usage_percent
Histogram Samples observations (usually duration/size) into configurable buckets. http_request_duration_seconds_bucket (calculating p95, p99 latency)
Summary Similar to Histogram, calculates streaming quantiles on the client side. High-frequency RPC latencies
# 1. Real-time request rate per second over the last 5 minutes
sum(rate(http_requests_total[5m])) by (status)
# 2. 99th percentile (p99) API response latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# 3. Memory pressure alert: percentage of free memory on Linux nodes
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10

Distributed Tracing: OpenTelemetry & Spans

Section titled “Distributed Tracing: OpenTelemetry & Spans”

In microservice architectures, a single user click might travel through an API Gateway, Auth Service, Billing Service, and Database. Distributed tracing propagates a Trace ID across HTTP headers (traceparent), tracking execution duration at each step:

[User Request] ──(Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736)──
│
├─► [API Gateway Ingress] (Span 1: 15ms)
├─► [Authentication Pod] (Span 2: 45ms)
├─► [Billing Database] (Span 3: 210ms - BOTTLENECK!)
└─► [Notification Worker] (Span 4: 10ms)

The 4 Golden Signals of SRE (Google SRE Framework)

Section titled “The 4 Golden Signals of SRE (Google SRE Framework)”

Every production dashboard and alert must be structured around Google’s Four Golden Signals:

  1. Latency: The time it takes to service a request (differentiate successful request latency from error latency).
  2. Traffic: A measure of how much demand is placed on your system (HTTP req/sec, network I/O).
  3. Errors: The rate of requests that fail (HTTP 500 errors, broken database connections).
  4. Saturation: How “full” your service is (memory saturation, CPU thread starvation, connection pool exhaustion).