Day 13: Retrieval-Augmented Generation (RAG) — Chatting with Proprietary Documents

2026-10-13 · 7 min read

A language model only knows what it was trained on, and that stopped at a fixed date. It has never read your internal wiki, your tickets or this morning's incident report. Retrieval-Augmented Generation (RAG) is the architecture that lets it answer questions about your own, up-to-date documents, without retraining the model.

In plain terms: An open-book exam. Instead of making the student memorise the whole library, you let them look up the right pages first, then write the answer from those pages.

A language model is frozen at its training date and has never seen your private documents. RAG looks them up when a question arrives and gives them to the model, so no retraining is needed and updated documents change the answers.

RAG chains together the two previous lessons: embeddings give every paragraph a position by meaning, and a vector database finds the closest ones fast. Then the LLM does what it is best at, which is writing a clear answer.

RAG has three steps: ingest your documents by chunking, embedding and storing them, retrieve the most relevant paragraphs for each question, then generate an answer from that retrieved context.

Step 1: Ingestion

Ahead of time, your private documents are split into chunks (paragraph-sized pieces), each chunk is converted into an embedding, and the vectors are stored in a vector database. Chunk size is a real design decision. Chunks that are too big blur the meaning, and chunks that are too small lose the surrounding context.

Ingestion splits your documents into chunks, converts each chunk into an embedding, and stores the vectors in a vector database. Chunk size is a trade-off: large chunks blur meaning and tiny chunks lose context.

Step 2: Retrieval

When a user asks a question, that question is embedded as well. The vector database runs a similarity search and returns the top few most relevant paragraphs. Three is a common starting point, and you tune it for your documents.

Step 3: Augmented generation

The retrieved paragraphs are injected into the LLM's prompt, together with an instruction: "Answer the user's question using ONLY the following retrieved context." The model writes its answer from the supplied facts instead of from memory.

The retrieved paragraphs are injected into the prompt with an instruction to answer using only that context. The model then writes its answer from the supplied facts instead of from memory. The example text is illustrative.

Here is the whole loop in Python. The embed, search and llm functions stand in for your embedding model, your vector database and your LLM API, so you can plug in the ones you use.

def answer(question: str) -> str:
    # 2. Retrieval: embed the question and fetch the 3 closest chunks
    query_vector = embed(question)
    chunks = search(query_vector, top_k=3)

    # 3. Augmented generation: put the chunks in the prompt
    context = "\n".join(f"- {c}" for c in chunks)
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context does not contain the answer, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt)

# 1. Ingestion (run ahead of time, and again whenever documents change)
for chunk in split_into_chunks(load_documents()):
    store(embed(chunk), chunk)

Notice the "say you don't know" rule. It gives the model a safe way out when retrieval finds nothing useful. Tomorrow's lesson covers what happens when retrieval goes wrong, and how to measure it.

Coming Up Next

Day 14: RAG evaluation and hallucination guardrails.

#RAG #GenerativeAI #EnterpriseAI #SoftwareEngineering #AIImplementation