AI Intermediate: Infrastructure & Agentic SystemsDay 10 of 30 · Week 2: Kubernetes Foundation for MLOps
Course outline▾

Week 2 · Kubernetes Foundation for MLOps

  1. Day 8K3d and Lightweight Kubernetes — Running Multi-Node Clusters on 16 GB of RAM
  2. Day 9Kubernetes Core for Machine Learning — Pods, Deployments and Headless Services
  3. Day 10Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
  4. Day 11StatefulSets for Vector Databases: Deploying Milvus or Qdrant in Kubernetes
  5. Day 12Auto-Scaling AI Inference: Using Horizontal Pod Autoscalers (HPA) for Spiky API Traffic
  6. Day 13Helm Charts: Packaging, Versioning, and Updating AI Deployments
  7. Day 14Week 2 Capstone: Deploying a Highly Available Local AI Application on K3d

Week 3 · High-Performance Networking & Security Mesh

  1. Day 15AI Network Bottlenecks: Transitioning from iptables to eBPF
  2. Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
  3. Day 17Service Mesh 101: Installing Istio over Your Local Cluster
  4. Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
  5. Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
  6. Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
  7. Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry

Week 4 · Enterprise Agents & CI/CD Workflows

  1. Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
  2. Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
  3. Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
  4. Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
  5. Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
  6. Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
  7. Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
  8. Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
  9. Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Open the course page →

Day 10: Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores

2026-11-16 · 7 min read

Pods are ephemeral. If every restart downloads a 10 to 30 GB model, you saturate bandwidth, slow every rolling update and can face 10 to 15 minute cold starts. The fix is to keep the weights on storage that outlives the pod.

COLD START WITH AND WITHOUT SHARED WEIGHTS (20 GB MODEL, 100 MB/s)Download on every restart200 s before the first tokenMount a shared volumeseconds: weights are already on diskArithmetic: 20,000 MB ÷ 100 MB/s = 200 s per pod, and every pod repeats it.Many teams see 10 to 15 minute cold starts when bandwidth is shared. Measure yours.
FIG 1 · Downloading a large model on each restart saturates bandwidth and slows rolling updates. A volume that already holds the weights turns minutes into seconds.

Three layers of storage

  • StorageClass: how storage is provisioned: a local path, AWS EBS, Ceph RBD and so on.
  • PersistentVolume (PV): a piece of storage provisioned by an administrator or dynamically.
  • PersistentVolumeClaim (PVC): the developer's request. It binds to a matching PV, and pods reference the claim.
THREE LAYERS OF KUBERNETES STORAGEStorageClassthe provisionerlocal path, AWS EBS, Ceph RBDPersistentVolumeadmin-provisioned storage50Gi · ReadOnlyMany · RetainPersistentVolumeClaimthe developer's requestshared-model-weights-pvcPod 1read-only mountPod 2read-only mountPod 3read-only mountPods reference the claim, never the disk.
FIG 2 · A StorageClass describes how storage is provisioned, a PersistentVolume is the storage itself and a PersistentVolumeClaim is the request that pods reference.

Access modes

  • ReadWriteOnce (RWO): read-write on a single node. Vector databases need it, because writes from several nodes can corrupt write-ahead-log segments.
  • ReadOnlyMany (ROX): read-only on many nodes at once. This is the standard way to share model weights: write them once, then mount them in dozens of stateless pods.
ACCESS MODES FOR AI WORKLOADSReadWriteOnce (RWO)one node reads and writesneeded for vector databases:multi-node writes corruptwrite-ahead-log segmentsone writer, one volumeReadOnlyMany (ROX)many nodes read at oncestandard for model weights:write once, mount acrossdozens of stateless podsone copy, many readersCheck that your storage backend supports ROX: network filesystems usually do.Local volumes need care, because each node has its own copy of a local path.
FIG 3 · Databases need ReadWriteOnce so only one writer touches the index. Model weights suit ReadOnlyMany, where many stateless pods read one copy.

Check that your storage backend supports the mode you ask for. Network filesystems such as NFS generally support ROX, while plain block disks usually do not.

Manifests for local weights

This example keeps weights in a local folder on the nodes.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-model-weights-pv
spec:
  capacity: { storage: 50Gi }
  volumeMode: Filesystem
  accessModes: [ReadOnlyMany]
  persistentVolumeReclaimPolicy: Retain          # keep the data if the claim is deleted
  storageClassName: local-storage
  local:
    path: /opt/ai/models
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: [k3d-ai-cluster-agent-0, k3d-ai-cluster-agent-1]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-model-weights-pvc
  namespace: ai-platform
spec:
  accessModes: [ReadOnlyMany]
  storageClassName: local-storage
  resources:
    requests: { storage: 50Gi }

A local volume is a path on one specific node, so each node needs its own copy of the files, and the nodeAffinity tells Kubernetes which nodes have them. For production, a shared network filesystem avoids copying the weights to every node.

The k3d gotcha

k3d nodes are Docker containers, so /opt/ai/models exists inside them only if you mount the host folder into every agent when you create the cluster. Without that, the PV points at an empty path.

LOCAL VOLUMES ON K3D: MAP THE FOLDER INTO THE NODESYour host/opt/ai/modelsk3d agent containers/opt/ai/models (mounted)PV local pathnodeAffinity: agent-0, agent-1Pods/models (read-only)k3d cluster create ai-cluster --agents 2 \ --volume /opt/ai/models:/opt/ai/models@agent:*Without the mount, the nodes cannot see the host folder, and the PV points at nothing.
FIG 4 · k3d nodes are containers, so a local PV path only exists on them if you mount the host folder into every agent when you create the cluster.
sudo mkdir -p /opt/ai/models
k3d cluster create ai-cluster --agents 2 \
  --volume /opt/ai/models:/opt/ai/models@agent:*

With the volume in place, a restarted pod mounts weights that are already on disk. Proper storage design removes repeated downloads, keeps rolling updates fast and protects your data across cluster changes.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster, using a hostPath volume instead of the lesson's local path. No pod mounts the volume here, so it only shows the binding.

apiVersion: v1
kind: PersistentVolume
metadata: {name: weights-pv}
spec:
  capacity: {storage: 50Gi}
  accessModes: [ReadOnlyMany]
  storageClassName: local-storage
  hostPath: {path: /tmp/ai-models}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: weights-pvc}
spec:
  accessModes: [ReadOnlyMany]
  storageClassName: local-storage
  resources: {requests: {storage: 50Gi}}

Run it:

kubectl apply -f storage.yaml

Output I got while recording the video:

pv  weights-pv Bound -> weights-pvc
pvc weights-pvc Bound -> weights-pv
size 50Gi, mode ReadOnlyMany

Coming Up Next

Day 11: StatefulSets for vector databases: deploying Milvus or Qdrant in Kubernetes.

#KubernetesStorage #PV #PVC #DataArchitecture #CloudNative #DevOps #StorageClass #MLOps