Day 16: Local Model Execution — Running Open-Weight LLMs Securely with Ollama and vLLM

2026-10-16 · 9 min read

Relying only on commercial cloud APIs carries real exposure: vendor lock-in, recurring cost for every token, rate limits, and the need to send sensitive intellectual property across public networks. Running an open-weight model such as Llama 3, Mistral or Gemma 2 on your own infrastructure addresses each of those. The price is that you now operate the hardware, so choosing the right runtime matters.

In plain terms: Ordering from a restaurant is easy, but you pay per dish, you queue at busy hours, and the restaurant sees what you ate. Cooking at home means buying the stove, and in return the menu, the speed and the privacy are yours.

Relying only on commercial cloud APIs brings vendor lock-in, recurring costs, rate limits and the transmission of sensitive data over public networks. Running open-weight models on your own infrastructure addresses each of these, at the price of operating the hardware yourself.

1. Prototyping and edge: Ollama and llama.cpp

On a workstation, a laptop or an edge box with limited memory (a 16 GB machine is a typical target), a plain Python script adds overhead you do not want. Ollama is built on llama.cpp, a lean C and C++ engine, so it avoids most of that Python overhead.

GGUF quantization: Ollama runs models stored in the GGUF format, usually at 4-bit or 8-bit precision. That shrinks multi-gigabyte weights enough to fit in ordinary RAM or a modest GPU, with a small loss in quality for most tasks. Check the quality on your own prompts.
Operational simplicity: it runs a local service and handles downloading models, spreading them across GPU and CPU memory, and formatting prompts. It exposes an OpenAI-compatible REST API at http://localhost:11434.

2. High-throughput serving: vLLM

When hundreds of users hit the same model at once, handling requests one after another creates a queue. vLLM is a serving engine built for that case, with two key ideas.

PagedAttention. Older engines reserve one contiguous block of KV-cache memory per request, sized for the longest possible reply, so much of it sits empty. The vLLM authors measured 60 to 80% waste in earlier systems. PagedAttention borrows the idea of virtual-memory paging from operating systems and splits the cache into small blocks that are handed out as a request grows.

Older inference engines reserve one contiguous block of KV-cache memory per request, sized for the maximum length, and much of it sits empty. vLLM's PagedAttention splits the cache into small blocks allocated as a request grows, like paging in an operating system. The vLLM authors reported 60 to 80% waste in earlier systems.

Continuous batching. A static batch must finish completely before new requests are accepted, so slots sit idle while the longest reply completes. vLLM slips new requests into the running batch at the token level. Reported gains are commonly 2x to 4x higher throughput than basic pipelines, but the real figure depends on your model, hardware and traffic.

With static batching, a whole batch must finish before new requests are accepted, so slots sit idle while the longest reply finishes. Continuous batching slips new requests into free slots at the token level, which commonly raises throughput by 2x to 4x depending on the model, hardware and traffic.

Which one should you pick?

Ollama, built on llama.cpp, suits prototyping and edge machines with limited memory and runs quantized GGUF models with a single command. vLLM suits GPU servers handling many users at once. Both expose an OpenAI-style API.

Here is how little your application code changes. Both engines speak an OpenAI-style API, so you can point the same client at either one. This assumes you have installed Ollama and pulled a model, or started a vLLM server.

# Ollama: pull and chat with a quantized 8B model in one command
ollama run llama3.1:8b

# vLLM: serve a model with an OpenAI-compatible API on port 8000 (needs an NVIDIA GPU)
vllm serve Qwen/Qwen2.5-7B-Instruct

from openai import OpenAI

# Same client, different base_url. Ollama listens on 11434, vLLM on 8000.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")

reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarise RAG in one sentence."}],
)
print(reply.choices[0].message.content)

Self-hosting turns an LLM from an outside cost center into a private utility inside your own network boundary. The next lessons package it properly, starting with containers.

Coming Up Next

Day 17: Containerizing AI workloads: writing production Dockerfiles for Python APIs.

#LocalAI #vLLM #Ollama #OpenSourceAI #DataSovereignty