Day 9: Kubernetes Core for Machine Learning — Pods, Deployments and Headless Services

A basic Deployment suits a stateless web server. Model serving and vector stores add failure domains that web apps rarely have: memory-exhaustion crashes, cold-boot timeouts and broken load-balancing rings. This lesson covers the Kubernetes building blocks that handle them.

Deployments and Services for inference

An inference pod ingests context, generates tokens and discards its KV-cache. It is stateless, so a Deployment gives you horizontal scaling, rolling updates and automatic restarts. A ClusterIP Service puts one stable address in front of the pods and sends traffic only to those that pass their readiness probe, so a saturated pod can step out of rotation without being killed.

Headless Services for peers

A normal Service gives one virtual IP. A headless Service (clusterIP: None) makes CoreDNS return one A record per pod, such as pod-0.db-svc and pod-1.db-svc. Distributed databases such as Qdrant and Milvus use that to find their peers.

A model-serving manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-engine-gemma
  namespace: ai-platform
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector:
    matchLabels: { app: inference-engine-gemma }
  template:
    metadata:
      labels: { app: inference-engine-gemma }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.5.0        # pin a tested tag, not "latest"
          args:
            - --model=/models/gemma-2-2b-it-awq
            - --quantization=awq
            - --max-model-len=4096
            - --gpu-memory-utilization=0.85
          ports: [{ name: http-api, containerPort: 8000 }]
          resources:
            requests: { cpu: "2000m", memory: 4Gi }
            limits:   { cpu: "4000m", memory: 8Gi }   # add nvidia.com/gpu: 1 on GPU nodes
          startupProbe:
            httpGet: { path: /health, port: http-api }
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 18                  # 15 + 18 x 10 = about 3 minutes to load
          readinessProbe:
            httpGet: { path: /v1/models, port: http-api }
            periodSeconds: 5
            timeoutSeconds: 2
          livenessProbe:
            httpGet: { path: /health, port: http-api }
            periodSeconds: 15
            failureThreshold: 3
          volumeMounts:
            - { name: model-weights, mountPath: /models, readOnly: true }
      volumes:
        - name: model-weights
          persistentVolumeClaim: { claimName: shared-model-weights-pvc }
---
apiVersion: v1
kind: Service
metadata:
  name: inference-engine-service
  namespace: ai-platform
spec:
  type: ClusterIP
  selector: { app: inference-engine-gemma }
  ports: [{ port: 8000, targetPort: http-api }]

Treat this as a structure to adapt. vLLM needs a GPU in practice, so on GPU nodes you would also request nvidia.com/gpu: 1 and run the NVIDIA device plugin, and the model files must exist on the volume you mount (Day 10).

Why the startupProbe is critical

Inference runtimes read gigabytes of weights before they open a network socket. If you rely on a livenessProbe alone, Kubernetes sees a silent port, concludes the pod is dead and restarts it, over and over. The startupProbe holds off the other probes until the server first passes, and here it allows about 3 minutes.

After startup, the readinessProbe decides whether the pod receives traffic, and the livenessProbe catches real hangs.

Rolling updates without a capacity dip

With maxSurge: 1 and maxUnavailable: 0, Kubernetes starts a new pod, waits for it to become ready, and only then retires an old one.

Together these settings turn a fragile model process into a service that restarts, scales and updates without dropping requests.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for the vLLM image so the pods start in seconds. The video wraps the command in a throwaway namespace and cleans up afterwards.

apiVersion: apps/v1
kind: Deployment
metadata: {name: vllm}
spec:
  replicas: 2
  selector: {matchLabels: {app: vllm}}
  strategy: {rollingUpdate: {maxSurge: 1, maxUnavailable: 0}}
  template:
    metadata: {labels: {app: vllm}}
    spec:
      containers:
        - name: vllm
          image: nginx:1.27-alpine
          startupProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 2
            failureThreshold: 30

Run it:

kubectl apply -f deploy.yaml

Output I got while recording the video:

vllm ready: 2/2
startup budget: 60 s
rolling update: done
ready pods: 2
revisions: 2

Coming Up Next

Day 10: Persistent volumes (PV and PVC): managing heavy model weights and vector stores.

#Kubernetes #K8sDeployments #MLOps #SystemDesign #vLLM #PlatformEngineering #HighAvailability