AI Intermediate: Infrastructure & Agentic SystemsDay 15 of 30 · Week 3: High-Performance Networking & Security Mesh
Course outline▾

Week 3 · High-Performance Networking & Security Mesh

  1. Day 15AI Network Bottlenecks — Transitioning from iptables to eBPF
  2. Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
  3. Day 17Service Mesh 101: Installing Istio over Your Local Cluster
  4. Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
  5. Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
  6. Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
  7. Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry

Week 4 · Enterprise Agents & CI/CD Workflows

  1. Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
  2. Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
  3. Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
  4. Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
  5. Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
  6. Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
  7. Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
  8. Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
  9. Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Open the course page →

Day 15: AI Network Bottlenecks — Transitioning from iptables to eBPF

2026-11-21 · 8 min read

Week 3 turns to the network and security layer. Most teams tune the model and the GPU, then discover that the network path inside the cluster is eating CPU that inference needed. This lesson explains why, and what replaces it.

How a Service IP reaches a pod

When one pod calls a Kubernetes Service, kube-proxy translates the Service IP to a pod IP using Linux iptables rules. Rules are appended to chains, and the kernel checks a packet against them in sequence. That is an O(N) lookup. In a cluster with thousands of Services the evaluation adds measurable latency to every new connection, and burns host CPU. (kube-proxy also has an IPVS mode that uses hash tables, but many clusters still run the default iptables mode.)

TWO WAYS A PACKET REACHES ANOTHER PODLEGACY: kube-proxy + iptablesAI poduser spacekube-proxyiptables chaindest podeBPF: a kernel hook and one hash lookupAI podeBPF hook in the kerneldest poddirectFewer hops, no trip through user space.
FIG 1 · With kube-proxy a packet is matched against iptables rules. An eBPF program in the kernel can map the destination directly, skipping those hops.
RULE LOOKUPS PER PACKET AS SERVICES GROW (ILLUSTRATIVE)010002000300040005000rules checkedservices in the clusteriptables: grows with NeBPF map: constant
FIG 2 · iptables evaluates rules in sequence, so cost grows with the number of services. A kernel hash map costs about the same however many services exist. The chart shows the shape, not measured numbers.

Why AI makes it worse

AI traffic is heavy. A request can carry a long context, and a RAG pipeline can stream batches of embeddings. A single 1,536-dimension float32 embedding is about 6 KB, a 128,000-token context is roughly half a megabyte of text, and a batch of 1,000 embeddings is around 6 MB. Multiply that by thousands of concurrent requests across replicas and vector databases, and packet handling competes with inference for the same CPUs.

AI PAYLOADS ARE HEAVY (APPROXIMATE)A chat message0.5 KBOne 1,536-dim embedding (float32)6 KB128k-token context (about 4 bytes/token)≈ 0.5 MB1,000 embeddings in one batch≈ 6 MBBars use a log scale. Big payloads plus thousands of requests make packet handling costly.When the host spends cycles on network rules, there is less left for inference.
FIG 3 · Embeddings and long contexts are far larger than ordinary web requests. Sizes are approximate: 1,536 float32 values is about 6 KB, and 128,000 tokens is roughly half a megabyte of text.

eBPF: programs in the kernel

eBPF lets you load small, verified programs into the Linux kernel's execution path without writing a kernel module or rebooting. Instead of walking an iptables chain, an eBPF program on a kernel hook looks up the destination in a hash map, a constant-time operation that does not depend on how many Services exist, and sends the packet straight to the target pod. It also avoids extra trips through user space.

HOW EBPF STAYS SAFE INSIDE THE KERNELWriterestricted C programCompileto eBPF bytecodeVerifykernel proves it is safeJITnative machine codeAttachhook in the data pathThe verifier rejects programs that could crash the kernel or loop forever.No kernel module, no reboot: programs load and update while traffic flows.
FIG 4 · An eBPF program is checked by the kernel verifier before it runs, which makes it safe to load into the data path without a custom kernel module.

The safety comes from the verifier: before an eBPF program runs, the kernel checks that it cannot crash the kernel, access memory it should not, or loop forever. That is what makes it acceptable to run programs from a networking plugin inside the kernel data path.

What to take away

  • The iptables model is O(N) in the number of services and can become a CPU tax for chatty AI workloads. Whether it hurts you depends on your cluster size and traffic, so measure before you migrate, for example by watching host CPU softirq time and per-connection latency.
  • eBPF replaces the rule chain with a constant-time lookup, which scales with traffic instead of with service count.
  • You do not write the eBPF programs yourself. A networking plugin such as Cilium ships them, and that is the next lesson.

You can see how many iptables rules a node carries with sudo iptables-save | wc -l. Compare that count on a small cluster and on a large one, and the growth with the number of Services becomes concrete.

Run it yourself

This is the exact example from the video. This counts steps for a rule chain and a hash lookup. It models the algorithms, it is not a benchmark of iptables or eBPF.

def chain(rules, key):          # iptables: walk the rules in order
    for n, r in enumerate(rules, 1):
        if r == key:
            return n

for n in (100, 1000, 10000):
    rules = [f"svc-{i}" for i in range(n)]
    table = {r: i for i, r in enumerate(rules)}   # eBPF: a hash map
    last = rules[-1]
    table[last]                                    # one lookup
    print(f"{n:5d} svcs: chain {chain(rules, last):5d}, map 1")

Run it:

python scan.py

Output I got while recording the video:

  100 svcs: chain   100, map 1
 1000 svcs: chain  1000, map 1
10000 svcs: chain 10000, map 1

Coming Up Next

Day 16: Cilium and eBPF: sub-millisecond routing for heavy AI microservices.

#eBPF #KubernetesNetworking #MLOps #LinuxKernel #PlatformEngineering #TechInfrastructure