Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware

A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup.

The acceleration architecture

A container cannot talk to a PCIe device without driver support. The NVIDIA Container Toolkit supplies it by wrapping the standard runtime with a hardware-aware one (nvidia-container-runtime).

Three ideas make it work.

1. Driver decoupling. The host runs the kernel driver (nvidia.ko). The image never contains it.
2. Runtime injection. At launch the toolkit injects the matching user-space driver library (libcuda.so) and the device nodes (/dev/nvidia*) into the container.
3. Application agility. The image needs only the high-level CUDA runtime libraries, so it does not depend on the host's driver build. The host driver must still be recent enough for the CUDA version inside the image.

Launching an accelerated container

With the toolkit installed, register it with Docker and restart the daemon.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

That edits /etc/docker/daemon.json. You can also set it by hand, and making it the default runtime is optional, since --gpus selects it per container.

{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }
  }
}

docker run -d --name local-inference \
  --gpus all \
  --shm-size="2g" \
  -p 8000:8000 \
  enterprise-model-server:latest

Verify it: docker run --rm --gpus all ubuntu nvidia-smi should print the same GPU table as the host.

Why --shm-size matters

PyTorch passes tensors between worker processes through shared memory (/dev/shm). Docker's default is only 64 MB, so parallel data loading can crash suddenly with bus errors. Raising it, for example to 2g, avoids that.

Hard limits for a 16 GB machine

An unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux cgroups.

docker run -d --name constrained-model \
  --cpus="4.0" \
  --memory="10g" \
  --memory-swap="10g" \
  -p 11434:11434 \
  ollama/ollama:latest

• --memory="10g" caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache.
• --memory-swap="10g" (equal to --memory) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast.
• When the container exceeds its cap, the kernel's OOM killer stops that container, not the host. This is the default behaviour (--oom-kill-disable=false), so you do not need to set it. Never use --oom-kill-disable=true without a memory limit, because then a spike can hang the whole machine.
• Add --gpus only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you.

Treat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits.

Run it yourself

This is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown.

# no GPU on this machine, so limits only
docker run --rm --cpus=2 --memory=256m --memory-swap=256m \
  --shm-size=2g alpine:3.20 sh -c '
  cd /sys/fs/cgroup
  for f in memory.max memory.swap.max cpu.max; do
    echo "$f: $(cat $f)"
  done
  echo shm: $(df -h /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)'

docker run --rm --memory=64m --memory-swap=64m alpine:3.20 \
  sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr "\0" x)'
echo "over the cap: exit code $?"

Run it:

bash limits.sh

Output I got while recording the video:

memory.max: 268435456
memory.swap.max: 0
cpu.max: 200000 100000
shm: 2.0G
over the cap: exit code 137

Coming Up Next

Day 4: Optimising model weights: quantization basics with GGUF and AWQ.

#HardwareAcceleration #Docker #GPUs #DevOps #Linux #TechInfrastructure #CUDA #SystemAdministration