Skip to content
About Me
----

OpenTelemetry: Distributed Tracing & Instrumentation Architecture

Architect Key Takeaways

Production guide to OpenTelemetry (OTel) architecture, W3C TraceContext context propagation, OTel Collector pipeline configuration, span attributes, and Jaeger/Tempo distributed tracing.

In monolithic systems, debugging a failed request meant checking a single log file on a single server. In modern microservices architectures where a single user click traverses dozens of Kubernetes pods, AWS Lambda functions, and managed databases, traditional logging fails completely.

OpenTelemetry (OTel) is the CNCF industry standard for vendor-neutral telemetry. It provides a unified framework of APIs, SDKs, and tooling to generate, collect, and export Traces, Metrics, and Logs (the Three Pillars of Observability) across distributed cloud systems.

OpenTelemetry Distributed Tracing Architecture


Understanding the vocabulary of distributed tracing is essential:

  • Trace: Represents the complete end-to-end journey of a request as it moves through a distributed system. A trace is a directed acyclic graph (DAG) of Spans.
  • Span: The single fundamental unit of work within a trace (e.g., an HTTP handler execution, an RPC call, or an SQL query). Each span records a start time, end time, status code, and rich metadata.
  • Trace Context & W3C Traceparent: To connect spans across different servers, services inject and extract HTTP headers adhering to the W3C TraceContext standard:
    traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    β”‚ β”‚ β”‚ β”‚
    β”‚ └─ Trace ID (16 bytes) └─ Span ID └─ Trace Flags (Sampled)
    └─ Version

Applications can be instrumented automatically (zero-code bytecode injection via OTel agents) or manually for business-critical operations.

2.1 Python Manual Instrumentation with Context Propagation

Section titled β€œ2.1 Python Manual Instrumentation with Context Propagation”
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import requests
# Obtain tracer instance
tracer = trace.get_tracer("payment-service", "1.4.0")
def process_checkout(order_id: str, amount_usd: float):
# 1. Start a new child span
with tracer.start_as_current_span("process_checkout") as span:
span.set_attribute("app.order_id", order_id)
span.set_attribute("payment.amount", amount_usd)
try:
# 2. Perform database lookup
with tracer.start_as_current_span("db_verify_inventory"):
verify_inventory(order_id)
# 3. Call downstream service with automatic trace context propagation
with tracer.start_as_current_span("http_stripe_charge") as http_span:
response = requests.post(
"https://api.stripe.com/v1/charges",
data={"amount": int(amount_usd * 100), "currency": "usd"},
timeout=5
)
http_span.set_attribute("http.status_code", response.status_code)
span.set_status(Status(StatusCode.OK))
return {"status": "success"}
except Exception as e:
# Record error event and fail the span
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
raise

The OTel Collector acts as a centralized telemetry proxy. Instead of having applications send data directly to Datadog, Jaeger, or New Relic, applications push to the local Collector via the OTLP (OpenTelemetry Protocol). The Collector then buffers, sanitizes, and fans out telemetry.

A collector pipeline consists of:

  1. Receivers: Ingest telemetry (OTLP gRPC port 4317, HTTP port 4318).
  2. Processors: Transform, filter, and batch data (batching, PII redaction, memory limits).
  3. Exporters: Translate and transmit data to storage backends (Jaeger, Tempo, Prometheus).

3.1 Production Collector Configuration (otel-collector-config.yaml)

Section titled β€œ3.1 Production Collector Configuration (otel-collector-config.yaml)”
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
# 1. Memory Limiter: Prevents Collector OOM crashing under heavy traffic spikes
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 20
# 2. Batch Processor: Groups spans and compresses with gzip for network efficiency
batch:
send_batch_size: 8192
timeout: 5s
# 3. K8s Attributes: Dynamically enriches spans with pod_name, namespace, and node
k8sattributes:
auth_type: "serviceAccount"
passthrough: false
extract:
metadata:
- k8s.pod.name
- k8s.namespace.name
- k8s.node.name
# 4. Redaction: Scrubs sensitive credit card numbers or API keys before export
redaction:
allow_all_keys: false
blocked_values:
- "(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14})" # Regex for credit cards
exporters:
# Export traces to Grafana Tempo / Jaeger
otlp/tempo:
endpoint: "tempo-distributor.monitoring.svc.cluster.local:4317"
tls:
insecure: true
# Export metrics to Prometheus
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "otel"
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, redaction, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]

The ultimate superpower of modern observability is Exemplars. When looking at a Prometheus graph showing a 99th percentile (p99) latency spike from 50ms to 4,000ms:

  • An Exemplar embeds the specific trace_id directly inside that metric data point.
  • In Grafana, clicking on the spike immediately opens the exact waterfall trace in Tempo/Jaeger, pinpointing the single slow database query or locked microservice thread in seconds.

  1. Deploy Collector as an Agent DaemonSet: Run an OTel Collector instance on every Kubernetes node (DaemonSet) listening on localhost:4317. Applications offload traces locally over loopback without network hops.
  2. Implement Tail-Based Sampling: Never sample 100% of traces in high-volume systems (too expensive). Use Collector tail-based sampling to retain 100% of error traces (http.status_code >= 500) and slow traces (duration > 1s), while sampling only 1% of normal 200 OK responses.
  3. Standardize Semantic Conventions: Always adhere to OpenTelemetry Semantic Conventions (e.g., http.method, http.status_code, db.system) so dashboards and alerts work universally across all languages.