Day 18: Docker Compose for Multi-Container AI Stacks

A serious AI application is a small distributed system. A RAG pipeline cannot be one monolithic process, because inference, data indexing and the user interface have different needs and change at different speeds. Docker Compose lets you describe all of them in one file and start them together on a single host.

In plain terms: Compose is a seating plan for a dinner party. You say who sits where, who can talk to whom, and who must arrive before the others, and Docker sets the table.

The core topology: a 3-tier RAG stack

• Model server (vLLM or Ollama). A dedicated compute container that manages GPU or system memory and exposes inference over an internal port.
• Vector database (Qdrant, Milvus or pgvector). An independent, stateful service that holds the embeddings and answers nearest-neighbour queries.
• Application layer (FastAPI, plus Streamlit or Next.js). The orchestrator: it takes the user's question, fetches matching context from the vector database, builds the prompt and asks the model server for the answer.

Three configuration practices

1. A private bridge network. Put every container on one user-defined network. Docker then resolves service names automatically, so the app can call http://ollama:11434 or http://qdrant:6333. Publish only the app's port to the host, and the database and model server stay unreachable from outside.

2. Volumes for state. Mount separate volumes for the vector database storage and the model cache (/root/.ollama, or /root/.cache/huggingface for Hugging Face models). Containers can then be updated, restarted or rebuilt without losing the index or downloading the weights again.

3. Healthchecks and start order. Add a healthcheck to each service and use depends_on with condition: service_healthy, so the app never queries a database or model server that is not listening yet.

Here is a compact example. It assumes your app lives in ./app with a Dockerfile like yesterday's, and that the healthcheck commands suit the image versions you use. Check each image's documentation, since minimal images do not always include the tools a check needs. In production, pin a specific image tag instead of latest.

services:
  app:
    build: ./app
    ports:
      - "8000:8000"            # the only port published to the host
    environment:
      OLLAMA_URL: http://ollama:11434
      QDRANT_URL: http://qdrant:6333
    depends_on:
      ollama:
        condition: service_healthy
      qdrant:
        condition: service_healthy
    networks: [ai-network]

  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant_data:/qdrant/storage
    healthcheck:
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 10s
      retries: 10
    networks: [ai-network]

  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama_models:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      retries: 10
    networks: [ai-network]
    # For an NVIDIA GPU, add a device reservation:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  qdrant_data:
  ollama_models:

networks:
  ai-network:

Start it with docker compose up -d, and stop it with docker compose down (add -v only if you want the volumes deleted too). Decoupling the stack this way means any one component can be upgraded, scaled or replaced without destabilizing the rest.

Coming Up Next

Day 19: Kubernetes for AI 101: pods, deployments and services for model serving.

#DockerCompose #Microservices #RAG #SystemDesign #DevOps