Course outline▾
Week 1 · Bare-Metal Infrastructure & Advanced Containerization
- Day 1The AI Infrastructure Shift — Why the Model is Only 10% of the Production System
- Day 2Advanced Docker for AI — Multi-Stage Builds for Heavy Python Runtimes
- Day 3Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware
- Day 4Optimizing Model Weights — Quantization Basics (GGUF, AWQ) for Efficient Inference
- Day 5Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution
- Day 6Containerizing AI Microservices — Building and Packaging APIs with FastAPI
- Day 7Docker Compose — Architecting a Multi-Container Database and Inference Stack Locally
Week 2 · Kubernetes Foundation for MLOps
- Day 8K3d and Lightweight Kubernetes — Running Multi-Node Clusters on 16 GB of RAM
- Day 9Kubernetes Core for Machine Learning — Pods, Deployments and Headless Services
- Day 10Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
- Day 11StatefulSets for Vector Databases: Deploying Milvus or Qdrant in Kubernetes
- Day 12Auto-Scaling AI Inference: Using Horizontal Pod Autoscalers (HPA) for Spiky API Traffic
- Day 13Helm Charts: Packaging, Versioning, and Updating AI Deployments
- Day 14Week 2 Capstone: Deploying a Highly Available Local AI Application on K3d
Week 3 · High-Performance Networking & Security Mesh
- Day 15AI Network Bottlenecks: Transitioning from iptables to eBPF
- Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
- Day 17Service Mesh 101: Installing Istio over Your Local Cluster
- Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
- Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
- Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
- Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry
Week 4 · Enterprise Agents & CI/CD Workflows
- Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
- Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
- Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
- Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
- Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
- Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
- Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
- Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
- Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Day 10: Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
2026-11-16 · 7 min read
Pods are ephemeral. If every restart downloads a 10 to 30 GB model, you saturate bandwidth, slow every rolling update and can face 10 to 15 minute cold starts. The fix is to keep the weights on storage that outlives the pod.
Three layers of storage
- StorageClass: how storage is provisioned: a local path, AWS EBS, Ceph RBD and so on.
- PersistentVolume (PV): a piece of storage provisioned by an administrator or dynamically.
- PersistentVolumeClaim (PVC): the developer's request. It binds to a matching PV, and pods reference the claim.
Access modes
- ReadWriteOnce (RWO): read-write on a single node. Vector databases need it, because writes from several nodes can corrupt write-ahead-log segments.
- ReadOnlyMany (ROX): read-only on many nodes at once. This is the standard way to share model weights: write them once, then mount them in dozens of stateless pods.
Check that your storage backend supports the mode you ask for. Network filesystems such as NFS generally support ROX, while plain block disks usually do not.
Manifests for local weights
This example keeps weights in a local folder on the nodes.
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-model-weights-pv
spec:
capacity: { storage: 50Gi }
volumeMode: Filesystem
accessModes: [ReadOnlyMany]
persistentVolumeReclaimPolicy: Retain # keep the data if the claim is deleted
storageClassName: local-storage
local:
path: /opt/ai/models
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: [k3d-ai-cluster-agent-0, k3d-ai-cluster-agent-1]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-model-weights-pvc
namespace: ai-platform
spec:
accessModes: [ReadOnlyMany]
storageClassName: local-storage
resources:
requests: { storage: 50Gi }
A local volume is a path on one specific node, so each node needs its own copy of the files, and the nodeAffinity tells Kubernetes which nodes have them. For production, a shared network filesystem avoids copying the weights to every node.
The k3d gotcha
k3d nodes are Docker containers, so /opt/ai/models exists inside them only if you mount the host folder into every agent when you create the cluster. Without that, the PV points at an empty path.
sudo mkdir -p /opt/ai/models
k3d cluster create ai-cluster --agents 2 \
--volume /opt/ai/models:/opt/ai/models@agent:*
With the volume in place, a restarted pod mounts weights that are already on disk. Proper storage design removes repeated downloads, keeps rolling updates fast and protects your data across cluster changes.
Run it yourself
This is the exact example from the video. Run against a real K3d cluster, using a hostPath volume instead of the lesson's local path. No pod mounts the volume here, so it only shows the binding.
apiVersion: v1
kind: PersistentVolume
metadata: {name: weights-pv}
spec:
capacity: {storage: 50Gi}
accessModes: [ReadOnlyMany]
storageClassName: local-storage
hostPath: {path: /tmp/ai-models}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: weights-pvc}
spec:
accessModes: [ReadOnlyMany]
storageClassName: local-storage
resources: {requests: {storage: 50Gi}}
Run it:
kubectl apply -f storage.yaml
Output I got while recording the video:
pv weights-pv Bound -> weights-pvc
pvc weights-pvc Bound -> weights-pv
size 50Gi, mode ReadOnlyMany
Coming Up Next
Day 11: StatefulSets for vector databases: deploying Milvus or Qdrant in Kubernetes.
#KubernetesStorage #PV #PVC #DataArchitecture #CloudNative #DevOps #StorageClass #MLOps