Cilium eBPF: Next-Generation Kubernetes Networking & Security
In-depth guide to Cilium eBPF architecture, completely replacing kube-proxy iptables, achieving constant O(1) service routing, Hubble network flow observability, and Layer 7 CiliumNetworkPolicies.
Kubernetes networking has traditionally relied on kube-proxy and the Linux kernel’s iptables or IPVS subsystems. While effective for small clusters, large-scale production deployments face catastrophic performance bottlenecks: iptables evaluates rules sequentially in $O(N)$ complexity, creating lock contention and CPU spikes whenever thousands of services are updated.
Cilium revolutionizes cloud-native networking by replacing iptables with eBPF (Extended Berkeley Packet Filter). eBPF executes verified bytecode directly inside the Linux kernel at the socket, traffic control (tc), and network driver (XDP) layers, achieving constant $O(1)$ lookup times and microsecond packet routing.
1. Why Replace Kube-Proxy with Cilium eBPF?
Section titled “1. Why Replace Kube-Proxy with Cilium eBPF?”| Performance Metric | Kube-Proxy (iptables) | Cilium eBPF (kubeProxyReplacement) |
|---|---|---|
| Lookup Algorithm | Linear sequential traversal: $O(N)$ | In-kernel BPF Hash Tables: $O(1)$ constant time |
| Scaling Limit | Noticeable latency degradation at >5,000 services | Millions of endpoints with zero packet latency degradation |
| Rule Updates | Full table lock and rewrite on every endpoint change | Atomic in-place updates of eBPF maps |
| Network Visibility | Layer 3/4 only (IP & Ports); zero Layer 7 visibility | Deep Layer 7 awareness (HTTP verbs, paths, DNS, Kafka) via Hubble |
| Client Source IP | Often masked by SNAT in NodePort traffic | Preserves true client IP addresses without complex hairpins |
| Encryption Overhead | Requires user-space proxies (Envoy sidecars) | In-kernel transparent WireGuard or IPsec encryption |
2. Production Installation: Cilium with Kube-Proxy Replacement
Section titled “2. Production Installation: Cilium with Kube-Proxy Replacement”To achieve maximum performance, configure Cilium to completely replace kube-proxy on your Kubernetes nodes:
helm repo add cilium https://helm.cilium.io/helm repo update
helm install cilium cilium/cilium \ --version 1.15.5 \ --namespace kube-system \ --set kubeProxyReplacement=true \ --set k8sServiceHost="api-server.internal" \ --set k8sServicePort=6443 \ --set cni.chainingMode=none \ --set bpf.masquerade=true \ --set tunnel=disabled \ --set autoDirectNodeRoutes=true \ --set routingMode=native \ --set hubble.enabled=true \ --set hubble.relay.enabled=true \ --set hubble.ui.enabled=true \ --set encryption.enabled=true \ --set encryption.type=wireguard3. Advanced CiliumNetworkPolicy: Layer 7 Security & FQDN Egress
Section titled “3. Advanced CiliumNetworkPolicy: Layer 7 Security & FQDN Egress”Standard Kubernetes NetworkPolicy objects can only filter traffic based on IP subnets and TCP/UDP ports. Cilium introduces CiliumNetworkPolicy, allowing security engineers to enforce Layer 7 HTTP rules and Fully Qualified Domain Name (FQDN) egress rules.
3.1 Layer 7 HTTP Method & Path Restrictions
Section titled “3.1 Layer 7 HTTP Method & Path Restrictions”Enforce that the frontend pod can only perform HTTP GET requests on public health endpoints, blocking malicious POST or DELETE attempts at the kernel level:
apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata: name: restrict-payments-api namespace: productionspec: endpointSelector: matchLabels: app: payments-service ingress: - fromEndpoints: - matchLabels: app: web-frontend toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "^/api/v1/orders/[0-9]+$" - method: "POST" path: "^/api/v1/checkout$"3.2 FQDN Egress Filtering: Protecting Cloud APIs
Section titled “3.2 FQDN Egress Filtering: Protecting Cloud APIs”Restricting outbound traffic to external SaaS APIs (like AWS S3 or Stripe) using static IPs is impossible because cloud providers use dynamic IP pools. Cilium intercepts DNS responses from CoreDNS and dynamically populates in-kernel eBPF ipsets:
apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata: name: allow-s3-and-stripe-egress namespace: productionspec: endpointSelector: matchLabels: app: invoice-generator egress: # 1. Allow DNS lookups so Cilium can intercept query responses - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: ANY rules: dns: - matchPattern: "*" # 2. Allow egress only to Stripe and AWS S3 - toFQDNs: - matchName: "api.stripe.com" - matchPattern: "*.s3.amazonaws.com" toPorts: - ports: - port: "443" protocol: TCP4. Real-Time Telemetry with Hubble
Section titled “4. Real-Time Telemetry with Hubble”Hubble is Cilium’s distributed observability platform built on top of eBPF. Because Hubble captures packets before they leave the Linux kernel, it adds zero user-space proxy overhead:
# Observe live HTTP traffic across the cluster in real timehubble observe --namespace production --protocol http
# Filter for dropped packets or policy denialshubble observe --verdict DROPPED
# Audit all DNS queries made by podshubble observe --protocol dns -f5. Architectural Summary & Best Practices
Section titled “5. Architectural Summary & Best Practices”- Enable Native Routing Mode: When running on cloud networks like AWS VPC or Azure VNet, use native routing (
routingMode=native) instead of Geneve/VXLAN encapsulation to eliminate packet MTU overhead. - Standardize on WireGuard: Enable transparent node-to-node encryption via in-kernel WireGuard (
encryption.type=wireguard) for compliance without the memory tax of sidecar service meshes. - Migrate Incrementally: Cilium can be introduced alongside existing CNI plugins (using CNI chaining mode) before converting to standalone native eBPF mode.