AI Intermediate: Infrastructure & Agentic SystemsDay 14 of 30 · Week 2: Kubernetes Foundation for MLOps
Course outline▾

Week 3 · High-Performance Networking & Security Mesh

  1. Day 15AI Network Bottlenecks: Transitioning from iptables to eBPF
  2. Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
  3. Day 17Service Mesh 101: Installing Istio over Your Local Cluster
  4. Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
  5. Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
  6. Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
  7. Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry

Week 4 · Enterprise Agents & CI/CD Workflows

  1. Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
  2. Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
  3. Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
  4. Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
  5. Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
  6. Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
  7. Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
  8. Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
  9. Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Open the course page →

Day 14: Week 2 Capstone — Deploying a Highly Available Local AI Application on K3d

2026-11-20 · 9 min read

This capstone ties Week 2 together. You will run a small but realistic platform on one machine: a multi-node cluster, persistent storage, a clustered vector database, scalable inference and a Helm release, then break it on purpose to see it heal.

The architecture

CAPSTONE ARCHITECTURE ON A 3-NODE K3D CLUSTERk3d-proxy:80IngressTraefikai-gatewayClusterIP :8000FastAPIorchestrator DeploymentStatefulSet qdrantqdrant-0 · qdrant-1vLLM Deploymentvllm-serviceembeddingstokensStorage· dedicated local PVC per Qdrant pod· ReadOnlyMany PVC for shared weightsEverything from Week 2: cluster, storage, StatefulSet, inference, Helm.
FIG 1 · Traffic enters through the k3d proxy and ingress, reaches a FastAPI orchestrator, which calls Qdrant for embeddings and vLLM for tokens. Each tier has its own storage.

Traffic enters through the k3d proxy on port 80 and reaches an ingress controller. The ingress sends it to the ai-gateway-service, a ClusterIP in front of a FastAPI orchestrator. The orchestrator calls the Qdrant StatefulSet (with its headless service and a dedicated PVC per pod) for embeddings, and the vLLM Deployment (with a ReadOnlyMany shared-weights volume) for tokens.

One choice to reconcile: this diagram uses an ingress controller, so do not disable Traefik for the capstone. Day 8 disabled it to make room for Envoy later, but here we use K3s's built-in Traefik so the stack works end to end.

The steps

FOUR STEPS TO DEPLOY1Create the clusterk3d cluster create ai-capstone --servers 1 --agents 2 -p "80:80@loadbalancer" --volume ...2Prepare host foldersmkdir /opt/ai/storage/qdrant and /models, owned by a normal user3Install the charthelm upgrade --install enterprise-ai ./helm-ai-stack -f values-staging.yaml4Verifykubectl get pods -n ai-system -o wide
FIG 2 · Create the cluster, prepare storage, install the Helm chart, then verify the pods. Keep the ingress controller enabled if your architecture depends on it.
# 1. Create the cluster (keep the built-in Traefik this time)
mkdir -p ~/ai-storage/qdrant ~/ai-storage/models
k3d cluster create ai-capstone --servers 1 --agents 2 \
  -p "80:80@loadbalancer" \
  --volume ~/ai-storage/models:/opt/ai/models@agent:*

# 2. Install the chart into its own namespace
helm upgrade --install enterprise-ai ./helm-ai-stack \
  --set global.namespace=ai-system --create-namespace \
  -f ./helm-ai-stack/values-staging.yaml

# 3. Check where everything landed
kubectl get nodes -o wide
kubectl get pods -n ai-system -o wide
kubectl get pvc -n ai-system

A note on permissions. You will often see sudo chmod -R 777 in tutorials to get a volume working. That is a lab shortcut that makes a folder writable by everyone, and you should not carry it into production. Prefer creating the folders as your own user, as above, or set the right owner and a tight mode such as 755.

If your machine has no GPU, replace the vLLM image in the chart with an Ollama-based one. The manifests keep the same shape, and you can still exercise every Kubernetes concept in the lesson.

Break it on purpose

Fault injection is how you learn whether the design holds.

kubectl delete pod -n ai-system -l app=local-ai-llm --now
kubectl get pods -n ai-system -w
FAULT INJECTION: DELETE A MODEL PODt = 0seconds laterpod servingdeletedreplacement startsready againkubectl delete pod -n ai-system -l app=local-ai-llm --nowThe Deployment controller notices and creates a new pod.The weights sit on a ReadOnlyMany volume, so nothing is re-downloaded.Recovery is dominated by model load time, not by network transfer.
FIG 3 · Deleting a model pod triggers an immediate replacement. Because the weights live on a shared volume, the new pod only has to load them, so it recovers far faster than it would after a re-download.

The Deployment controller notices the missing pod and creates a replacement at once. Because the weights live on a ReadOnlyMany volume that is already on the nodes, the new pod does not re-download anything. It passes its startup probe as soon as the model has loaded, which is much quicker than a fresh download.

Verify the whole system

WHAT TO CHECK BEFORE YOU CALL IT DONE✓Nodes Readykubectl get nodes✓Pods spread across agentskubectl get pods -o wide✓PVCs Boundkubectl get pvc -n ai-system✓Qdrant peers see each othercluster info on :6333✓HPA sees its metrickubectl get hpa✓Recovery after deleting a podtime to ReadyIf you have no GPU, swap vLLM for Ollama: the manifests keep the same shape.
FIG 4 · Verify nodes, pod placement, volume binding, database clustering, autoscaling and recovery. On a machine without a GPU, run a CPU-friendly engine such as Ollama in place of vLLM.
  1. Nodes show Ready.
  2. Pods are spread across the agents, with Qdrant and vLLM on different nodes.
  3. PVCs show Bound.
  4. Qdrant peers can see each other (check the cluster endpoint on port 6333).
  5. The HPA can read its metric.
  6. Deleting a pod recovers, and you can time how long that takes.

That completes Week 2. You now have a cluster you can create in seconds, storage that outlives pods, a database that keeps its identity, inference that scales on the right signal and a Helm chart that ships it all. Week 3 turns to the network and security layer: eBPF, Cilium and service meshes.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for the model server, and the video deletes one of the two pods and counts what comes back.

apiVersion: apps/v1
kind: Deployment
metadata: {name: llm}
spec:
  replicas: 2
  selector: {matchLabels: {app: llm}}
  template:
    metadata: {labels: {app: llm}}
    spec:
      containers:
        - name: llm
          image: nginx:1.27-alpine
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 1

Run it:

kubectl apply -f deploy.yaml
kubectl delete pod <one of the two pods>
kubectl get pods

Output I got while recording the video:

before: 2/2 ready
deleted: 1 pod
after: 2/2 ready
new pods created: 1
deleted pod gone: yes

Coming Up Next

Day 15: AI network bottlenecks: moving from iptables to eBPF.

#Kubernetes #MLOps #SystemDesign #CapStone #CloudNative #DevOps #DataScience #AIEngineering