OpenTelemetry: Distributed Tracing & Instrumentation Architecture
Production guide to OpenTelemetry (OTel) architecture, W3C TraceContext context propagation, OTel Collector pipeline configuration, span attributes, and Jaeger/Tempo distributed tracing.
In monolithic systems, debugging a failed request meant checking a single log file on a single server. In modern microservices architectures where a single user click traverses dozens of Kubernetes pods, AWS Lambda functions, and managed databases, traditional logging fails completely.
OpenTelemetry (OTel) is the CNCF industry standard for vendor-neutral telemetry. It provides a unified framework of APIs, SDKs, and tooling to generate, collect, and export Traces, Metrics, and Logs (the Three Pillars of Observability) across distributed cloud systems.
1. Core Distributed Tracing Concepts
Section titled β1. Core Distributed Tracing ConceptsβUnderstanding the vocabulary of distributed tracing is essential:
- Trace: Represents the complete end-to-end journey of a request as it moves through a distributed system. A trace is a directed acyclic graph (DAG) of Spans.
- Span: The single fundamental unit of work within a trace (e.g., an HTTP handler execution, an RPC call, or an SQL query). Each span records a start time, end time, status code, and rich metadata.
- Trace Context & W3C Traceparent: To connect spans across different servers, services inject and extract HTTP headers adhering to the W3C TraceContext standard:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01β β β ββ ββ Trace ID (16 bytes) ββ Span ID ββ Trace Flags (Sampled)ββ Version
2. In-App Instrumentation: Python & Go
Section titled β2. In-App Instrumentation: Python & GoβApplications can be instrumented automatically (zero-code bytecode injection via OTel agents) or manually for business-critical operations.
2.1 Python Manual Instrumentation with Context Propagation
Section titled β2.1 Python Manual Instrumentation with Context Propagationβfrom opentelemetry import tracefrom opentelemetry.trace import Status, StatusCodeimport requests
# Obtain tracer instancetracer = trace.get_tracer("payment-service", "1.4.0")
def process_checkout(order_id: str, amount_usd: float): # 1. Start a new child span with tracer.start_as_current_span("process_checkout") as span: span.set_attribute("app.order_id", order_id) span.set_attribute("payment.amount", amount_usd)
try: # 2. Perform database lookup with tracer.start_as_current_span("db_verify_inventory"): verify_inventory(order_id)
# 3. Call downstream service with automatic trace context propagation with tracer.start_as_current_span("http_stripe_charge") as http_span: response = requests.post( "https://api.stripe.com/v1/charges", data={"amount": int(amount_usd * 100), "currency": "usd"}, timeout=5 ) http_span.set_attribute("http.status_code", response.status_code)
span.set_status(Status(StatusCode.OK)) return {"status": "success"}
except Exception as e: # Record error event and fail the span span.record_exception(e) span.set_status(Status(StatusCode.ERROR, str(e))) raise3. The OpenTelemetry Collector: Production Pipeline
Section titled β3. The OpenTelemetry Collector: Production PipelineβThe OTel Collector acts as a centralized telemetry proxy. Instead of having applications send data directly to Datadog, Jaeger, or New Relic, applications push to the local Collector via the OTLP (OpenTelemetry Protocol). The Collector then buffers, sanitizes, and fans out telemetry.
A collector pipeline consists of:
- Receivers: Ingest telemetry (OTLP gRPC port
4317, HTTP port4318). - Processors: Transform, filter, and batch data (batching, PII redaction, memory limits).
- Exporters: Translate and transmit data to storage backends (Jaeger, Tempo, Prometheus).
3.1 Production Collector Configuration (otel-collector-config.yaml)
Section titled β3.1 Production Collector Configuration (otel-collector-config.yaml)βreceivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318
processors: # 1. Memory Limiter: Prevents Collector OOM crashing under heavy traffic spikes memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20
# 2. Batch Processor: Groups spans and compresses with gzip for network efficiency batch: send_batch_size: 8192 timeout: 5s
# 3. K8s Attributes: Dynamically enriches spans with pod_name, namespace, and node k8sattributes: auth_type: "serviceAccount" passthrough: false extract: metadata: - k8s.pod.name - k8s.namespace.name - k8s.node.name
# 4. Redaction: Scrubs sensitive credit card numbers or API keys before export redaction: allow_all_keys: false blocked_values: - "(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14})" # Regex for credit cards
exporters: # Export traces to Grafana Tempo / Jaeger otlp/tempo: endpoint: "tempo-distributor.monitoring.svc.cluster.local:4317" tls: insecure: true
# Export metrics to Prometheus prometheus: endpoint: "0.0.0.0:8889" namespace: "otel"
service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, k8sattributes, redaction, batch] exporters: [otlp/tempo] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [prometheus]4. Exemplars: Bridging Metrics to Traces
Section titled β4. Exemplars: Bridging Metrics to TracesβThe ultimate superpower of modern observability is Exemplars. When looking at a Prometheus graph showing a 99th percentile (p99) latency spike from 50ms to 4,000ms:
- An Exemplar embeds the specific
trace_iddirectly inside that metric data point. - In Grafana, clicking on the spike immediately opens the exact waterfall trace in Tempo/Jaeger, pinpointing the single slow database query or locked microservice thread in seconds.
5. SRE Best Practices for OpenTelemetry
Section titled β5. SRE Best Practices for OpenTelemetryβ- Deploy Collector as an Agent DaemonSet: Run an OTel Collector instance on every Kubernetes node (
DaemonSet) listening onlocalhost:4317. Applications offload traces locally over loopback without network hops. - Implement Tail-Based Sampling: Never sample 100% of traces in high-volume systems (too expensive). Use Collector tail-based sampling to retain 100% of error traces (
http.status_code >= 500) and slow traces (duration > 1s), while sampling only 1% of normal 200 OK responses. - Standardize Semantic Conventions: Always adhere to OpenTelemetry Semantic Conventions (e.g.,
http.method,http.status_code,db.system) so dashboards and alerts work universally across all languages.