Skip to content
About Me
----

Cilium eBPF: Next-Generation Kubernetes Networking & Security

Architect Key Takeaways

In-depth guide to Cilium eBPF architecture, completely replacing kube-proxy iptables, achieving constant O(1) service routing, Hubble network flow observability, and Layer 7 CiliumNetworkPolicies.

Kubernetes networking has traditionally relied on kube-proxy and the Linux kernel’s iptables or IPVS subsystems. While effective for small clusters, large-scale production deployments face catastrophic performance bottlenecks: iptables evaluates rules sequentially in $O(N)$ complexity, creating lock contention and CPU spikes whenever thousands of services are updated.

Cilium revolutionizes cloud-native networking by replacing iptables with eBPF (Extended Berkeley Packet Filter). eBPF executes verified bytecode directly inside the Linux kernel at the socket, traffic control (tc), and network driver (XDP) layers, achieving constant $O(1)$ lookup times and microsecond packet routing.

Cilium eBPF Architecture


1. Why Replace Kube-Proxy with Cilium eBPF?

Section titled “1. Why Replace Kube-Proxy with Cilium eBPF?”
Performance Metric Kube-Proxy (iptables) Cilium eBPF (kubeProxyReplacement)
Lookup Algorithm Linear sequential traversal: $O(N)$ In-kernel BPF Hash Tables: $O(1)$ constant time
Scaling Limit Noticeable latency degradation at >5,000 services Millions of endpoints with zero packet latency degradation
Rule Updates Full table lock and rewrite on every endpoint change Atomic in-place updates of eBPF maps
Network Visibility Layer 3/4 only (IP & Ports); zero Layer 7 visibility Deep Layer 7 awareness (HTTP verbs, paths, DNS, Kafka) via Hubble
Client Source IP Often masked by SNAT in NodePort traffic Preserves true client IP addresses without complex hairpins
Encryption Overhead Requires user-space proxies (Envoy sidecars) In-kernel transparent WireGuard or IPsec encryption

2. Production Installation: Cilium with Kube-Proxy Replacement

Section titled “2. Production Installation: Cilium with Kube-Proxy Replacement”

To achieve maximum performance, configure Cilium to completely replace kube-proxy on your Kubernetes nodes:

Terminal window
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium \
--version 1.15.5 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost="api-server.internal" \
--set k8sServicePort=6443 \
--set cni.chainingMode=none \
--set bpf.masquerade=true \
--set tunnel=disabled \
--set autoDirectNodeRoutes=true \
--set routingMode=native \
--set hubble.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set encryption.enabled=true \
--set encryption.type=wireguard

3. Advanced CiliumNetworkPolicy: Layer 7 Security & FQDN Egress

Section titled “3. Advanced CiliumNetworkPolicy: Layer 7 Security & FQDN Egress”

Standard Kubernetes NetworkPolicy objects can only filter traffic based on IP subnets and TCP/UDP ports. Cilium introduces CiliumNetworkPolicy, allowing security engineers to enforce Layer 7 HTTP rules and Fully Qualified Domain Name (FQDN) egress rules.

3.1 Layer 7 HTTP Method & Path Restrictions

Section titled “3.1 Layer 7 HTTP Method & Path Restrictions”

Enforce that the frontend pod can only perform HTTP GET requests on public health endpoints, blocking malicious POST or DELETE attempts at the kernel level:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: restrict-payments-api
namespace: production
spec:
endpointSelector:
matchLabels:
app: payments-service
ingress:
- fromEndpoints:
- matchLabels:
app: web-frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "^/api/v1/orders/[0-9]+$"
- method: "POST"
path: "^/api/v1/checkout$"

3.2 FQDN Egress Filtering: Protecting Cloud APIs

Section titled “3.2 FQDN Egress Filtering: Protecting Cloud APIs”

Restricting outbound traffic to external SaaS APIs (like AWS S3 or Stripe) using static IPs is impossible because cloud providers use dynamic IP pools. Cilium intercepts DNS responses from CoreDNS and dynamically populates in-kernel eBPF ipsets:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: allow-s3-and-stripe-egress
namespace: production
spec:
endpointSelector:
matchLabels:
app: invoice-generator
egress:
# 1. Allow DNS lookups so Cilium can intercept query responses
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
# 2. Allow egress only to Stripe and AWS S3
- toFQDNs:
- matchName: "api.stripe.com"
- matchPattern: "*.s3.amazonaws.com"
toPorts:
- ports:
- port: "443"
protocol: TCP

Hubble is Cilium’s distributed observability platform built on top of eBPF. Because Hubble captures packets before they leave the Linux kernel, it adds zero user-space proxy overhead:

Terminal window
# Observe live HTTP traffic across the cluster in real time
hubble observe --namespace production --protocol http
# Filter for dropped packets or policy denials
hubble observe --verdict DROPPED
# Audit all DNS queries made by pods
hubble observe --protocol dns -f

  1. Enable Native Routing Mode: When running on cloud networks like AWS VPC or Azure VNet, use native routing (routingMode=native) instead of Geneve/VXLAN encapsulation to eliminate packet MTU overhead.
  2. Standardize on WireGuard: Enable transparent node-to-node encryption via in-kernel WireGuard (encryption.type=wireguard) for compliance without the memory tax of sidecar service meshes.
  3. Migrate Incrementally: Cilium can be introduced alongside existing CNI plugins (using CNI chaining mode) before converting to standalone native eBPF mode.