AI Intermediate: Infrastructure & Agentic SystemsDay 11 of 30 · Week 2: Kubernetes Foundation for MLOps
Course outline▾

Week 2 · Kubernetes Foundation for MLOps

  1. Day 8K3d and Lightweight Kubernetes — Running Multi-Node Clusters on 16 GB of RAM
  2. Day 9Kubernetes Core for Machine Learning — Pods, Deployments and Headless Services
  3. Day 10Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
  4. Day 11StatefulSets for Vector Databases — Deploying Milvus or Qdrant in Kubernetes
  5. Day 12Auto-Scaling AI Inference: Using Horizontal Pod Autoscalers (HPA) for Spiky API Traffic
  6. Day 13Helm Charts: Packaging, Versioning, and Updating AI Deployments
  7. Day 14Week 2 Capstone: Deploying a Highly Available Local AI Application on K3d

Week 3 · High-Performance Networking & Security Mesh

  1. Day 15AI Network Bottlenecks: Transitioning from iptables to eBPF
  2. Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
  3. Day 17Service Mesh 101: Installing Istio over Your Local Cluster
  4. Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
  5. Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
  6. Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
  7. Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry

Week 4 · Enterprise Agents & CI/CD Workflows

  1. Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
  2. Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
  3. Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
  4. Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
  5. Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
  6. Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
  7. Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
  8. Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
  9. Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Open the course page →

Day 11: StatefulSets for Vector Databases — Deploying Milvus or Qdrant in Kubernetes

2026-11-17 · 8 min read

A crashed stateless pod comes back with a random name and a new IP, and nobody minds. A database node that returns with a different identity can break cluster consensus and force expensive re-sharding. A StatefulSet fixes that with stable network identities, ordered operations and automatic per-pod storage.

Identity and storage

STABLE IDENTITY, DEDICATED STORAGEqdrant-0.qdrant-headless.ai-platform.svcPVC data-qdrant-0own 20Gi volumeqdrant-1.qdrant-headless.ai-platform.svcPVC data-qdrant-1own 20Gi volumeqdrant-2.qdrant-headless.ai-platform.svcPVC data-qdrant-2own 20Gi volumeStartup order: qdrant-0 → qdrant-1 → qdrant-2. The name survives a restart.A stateless pod gets a random name. A database node must keep the same one.
FIG 1 · Each StatefulSet pod keeps a stable DNS name and its own PersistentVolumeClaim, and pods start in order. A restarted pod comes back as the same member.

Each pod gets a predictable DNS name, such as qdrant-0.qdrant-headless.ai-platform.svc.cluster.local, plus its own PVC. Pods start in order, qdrant-0 then qdrant-1, and the name survives a restart.

Why Deployments fail databases

THREE WAYS A DEPLOYMENT FAILS A DATABASEIDENTITY INSTABILITYrandom names and IPs after a restart force costly re-sync of high-dimensional indexesSTORAGE CONTENTIONa Deployment shares one claim, a StatefulSet uses volumeClaimTemplates: one volume perreplicaUNCONTROLLED TERMINATIONall pods stop at once, a StatefulSet stops in reverse order: 2, then 1, then 0
FIG 2 · Identity, storage and shutdown order all matter to a clustered database. StatefulSets handle each, and Deployments do not.
  1. Identity instability. Random names and IPs after a restart force the cluster to re-sync large indexes.
  2. Storage contention. A Deployment shares one claim among its pods. A StatefulSet uses volumeClaimTemplates to create a dedicated volume per replica.
  3. Uncontrolled termination. A Deployment can stop many pods at once. A StatefulSet stops them in strict reverse order, qdrant-2, then qdrant-1, then qdrant-0, which helps the cluster keep quorum.
ORDERED STARTUP, REVERSE SHUTDOWNSTARTUPSHUTDOWNqdrant-0qdrant-1qdrant-2qdrant-2qdrant-1qdrant-0Reverse order keeps a majority of nodes alive for as long as possible.That protects the Raft quorum while pods stop one at a time.
FIG 3 · Pods start from zero upward and stop from the highest ordinal downward, which helps clustered databases keep quorum during restarts and updates.

The manifest

This runs Qdrant in clustered mode. Qdrant needs each new peer to know where to bootstrap from, so the first pod starts the cluster and the others join it.

apiVersion: v1
kind: Service
metadata:
  name: qdrant-headless
  namespace: ai-platform
spec:
  clusterIP: None
  selector: { app: qdrant }
  ports:
    - { name: http, port: 6333 }
    - { name: internal-p2p, port: 6335 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
  namespace: ai-platform
spec:
  serviceName: qdrant-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  selector:
    matchLabels: { app: qdrant }
  template:
    metadata:
      labels: { app: qdrant }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.9.0
          env:
            - { name: QDRANT__CLUSTER__ENABLED, value: "true" }
            - { name: QDRANT__CLUSTER__P2P__PORT, value: "6335" }
          command: ["/bin/sh", "-c"]
          args:
            - |
              HOST="$(hostname).qdrant-headless"
              if [ "$(hostname)" = "qdrant-0" ]; then
                ./qdrant --uri "http://${HOST}:6335"
              else
                ./qdrant --bootstrap "http://qdrant-0.qdrant-headless:6335" --uri "http://${HOST}:6335"
              fi
          ports:
            - { name: http, containerPort: 6333 }
            - { name: p2p, containerPort: 6335 }
          resources:
            requests: { cpu: "1000m", memory: 2Gi }
            limits:   { cpu: "2000m", memory: 4Gi }
          volumeMounts:
            - { name: qdrant-data, mountPath: /qdrant/storage }
  volumeClaimTemplates:
    - metadata: { name: qdrant-data }
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests: { storage: 20Gi }

Check the bootstrap flags against the Qdrant version you run, because clustering options have changed between releases.

How many replicas?

Raft-based clusters need a majority of nodes to agree. One and two nodes tolerate no failures, three tolerate one, and five tolerate two. Two replicas are fine for a lab, but for real fault tolerance run three.

RAFT QUORUM: HOW MANY NODES CAN FAIL?1 nodetolerates 0 failures2 nodestolerates 0 failures3 nodestolerates 1 failure5 nodestolerates 2 failuresA majority must stay up. Two nodes give no more resilience than one.For real fault tolerance run three replicas. Two are fine for a lab.
FIG 4 · A Raft cluster needs a majority of nodes to agree. Three nodes tolerate one failure, five tolerate two, and two nodes tolerate none, so use at least three replicas when availability matters.

Proper StatefulSet design protects your vector indexes through restarts and upgrades, and it gives the rest of the stack a database that behaves predictably.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for Qdrant so three pods start quickly.

{apiVersion: v1, kind: Service, metadata: {name: qdrant-headless},
 spec: {clusterIP: None, selector: {app: qdrant}}}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: qdrant}
spec:
  serviceName: qdrant-headless
  replicas: 3
  selector: {matchLabels: {app: qdrant}}
  template:
    metadata: {labels: {app: qdrant}}
    spec: {containers: [{name: qdrant, image: "nginx:1.27-alpine"}]}
  volumeClaimTemplates:
    - metadata: {name: data}
      spec: {accessModes: [ReadWriteOnce],
        resources: {requests: {storage: 1Gi}}}

Run it:

kubectl apply -f statefulset.yaml

Output I got while recording the video:

qdrant-0 Running
qdrant-1 Running
qdrant-2 Running
data-qdrant-0 Bound
data-qdrant-1 Bound
data-qdrant-2 Bound
headless clusterIP: None

Coming Up Next

Day 12: Auto-scaling AI inference with Horizontal Pod Autoscalers for spiky API traffic.

#StatefulSet #VectorDatabase #Qdrant #Milvus #CloudNative #DataEngineering #SystemDesign