Day 15: Introduction to AI Infrastructure — Hardware, Runtimes and Compute

2026-10-15 · 7 min read

Until now we called AI over an API. Moving to self-hosted models is how you guarantee your data never leaves your network, and it also means you now own the hardware, the runtime and the operating system. This week starts with the three building blocks.

In plain terms: Hosting your own model is like running your own kitchen instead of ordering delivery. You get control and privacy, but you now need the right stove, the right pans and a kitchen that stays up.

1. The hardware constraint: VRAM

A model has to fit in the memory of the GPU, called VRAM, before it can run. The parameter count sets the requirement: memory is roughly the number of parameters times the bytes used per parameter. Shrinking the numbers (quantization) is how you fit bigger models on smaller cards.

The weights of an 8-billion-parameter model take about 16 GB at 16-bit precision, 8 GB at 8-bit and 4 GB at 4-bit. The KV cache and runtime add roughly 1 to 3 GB, so a 4-bit 8B model needs roughly 6 to 8 GB of VRAM in practice.

The figures are for the weights alone. The KV cache, which holds the conversation so far, and the runtime itself take extra space, so plan for a little headroom. A quantized 8-billion-parameter model needs roughly 6 to 8 GB of VRAM in practice.

2. Inference runtimes

You do not run a raw Python script in production. You use a dedicated, optimized runtime. llama.cpp is a lean C++ runtime that runs quantized models well on laptops and small servers. vLLM is built for GPU servers and serves many requests at once with smart batching and memory management. Same model, far more speed from the same hardware.

You do not run raw Python scripts in production. Dedicated runtimes such as llama.cpp and vLLM squeeze much more speed from limited hardware using optimized code, batching and efficient memory use.

3. The hosting OS

Linux is the standard host for production AI. A headless Ubuntu Server, which means no desktop environment at all, is a common choice. It leaves every bit of memory and CPU for your workloads, and it passes GPU resources cleanly into Docker containers and Kubernetes pods.

Headless Ubuntu Server has no desktop overhead, passes GPU drivers cleanly to Docker and Kubernetes, and offers long-term support releases that are easy to automate, which is why Linux is the standard host for production AI.

Put the pieces together and you get a stack that is built from the bottom up.

A self-hosted AI server is built bottom up: GPU hardware, a headless Ubuntu Server with the GPU driver, the NVIDIA container toolkit that exposes the GPU to containers, Docker or Kubernetes pods, and finally the model running in an optimized runtime.

Once the driver and the container toolkit are in place, you can check the GPU from inside a container. The commands below are the standard checks, and they need an NVIDIA GPU on the host.

# On the host: is the GPU visible to the driver?
nvidia-smi

# Inside a container: is the GPU passed through?
docker run --rm --gpus all ubuntu nvidia-smi

If the second command prints the same GPU table as the first, your containers can use the GPU. The next lessons build on that: local models with Ollama and vLLM, then Dockerfiles and Kubernetes.

Coming Up Next

Day 16: Local model execution: running open-weight LLMs securely with Ollama and vLLM.

#AIInfrastructure #Hardware #Ubuntu #Docker #SelfHostedAI