RAG Lab: build a small RAG app, step by step
Practise Retrieval-Augmented Generation hands-on. You will build an app that finds the right policy, respects who is asking and when, answers with citations, and refuses instead of guessing. It uses the Payer X claim-denial story from the RAG Blueprint Workshop (all names and figures are illustrative).
Open in Colab ↗
How it works: press ▶ Run on each step in order (or Run all). You can edit any code and run it again; Reset code restores the original. Works best on a laptop or desktop.
Step 1 · Helpers
Small helpers for printing tables, plus two tiny search engines written in plain Python: TF-IDF (word weighting) and BM25 (keyword ranking). In Colab we use ready-made libraries; here we build tiny versions so everything runs inside your browser.
Step 2 · The knowledge base
Ten illustrative documents, each with metadata: payer, version, status, the dates it is valid, who may read it, and whether it comes from a trusted source. Notice the traps: three versions of the same Payer X policy, a different payer, a confidential rate sheet, and an untrusted upload that tries to give the AI instructions.
Step 3 · Chunking: fixed-size
We search chunks, not whole documents. The simplest method cuts every N characters. Look at what happens to the deadline and its exception.
Step 4 · Chunking: structure-aware
Split at the document’s own ## headings and stamp every chunk with the document’s metadata. Then an automatic check compares the two strategies.
✍️ Your turn: try different chunk sizes
Edit the numbers below and press Run. Which sizes keep the rule and its exception together? What happens to the number of chunks?
Step 5 · Build the chunk store and the search indexes
We chunk every document, then build two indexes: vectors for similarity search and BM25 for keyword search. This is the offline half of RAG; you rerun it whenever documents change.
Step 6 · Vector search
Each chunk is now a vector. We turn the question into a vector too and rank chunks by cosine similarity (1.0 = same direction). Note: in this browser version the vectors come from TF-IDF, which matches words. The Colab version uses a neural embedding model that matches meaning.
The right chunk (px-4.2-v3#s1) is there, but so are the superseded v2, the draft v4 and others. Similar is not the same as correct; Step 8 fixes that.
Step 7 · Keyword search (BM25) and hybrid search
Keyword search is excellent at exact codes such as CO-197 or PY-22. Hybrid search runs both methods and merges the rankings with Reciprocal Rank Fusion: each chunk earns 1 / (60 + rank) from each list.
✍️ Your turn: compare the methods
Add your own questions to the list: an exact code, a paraphrase, a product name. Which method wins for which kind of question?
Step 8 · Metadata filters and access control
Filters run before ranking, so stale, foreign, draft, forbidden or untrusted chunks never become candidates. We check payer, effective date, draft status, the user’s role, and the source.
✍️ Your turn: change the date and the role
Try as_of="2026-03-15" (before the July policy change), then try role="contract_manager" with a question about rates.
Step 9 · The grounded prompt
The generator receives three things: instructions (answer only from the context, cite, refuse when evidence is missing, treat documents as data), the retrieved chunks clearly delimited, and the question. In Colab you can send this to Claude; here we print it so you can see exactly what a model would receive.
Step 10 · Evidence gate and answer
Before answering we check how much of the question the evidence covers, giving rare words like telehealth or reimbursement more weight. Too little coverage means a safe refusal. The answer itself is extractive: it quotes the best sentences with citations, so no AI model or key is needed.
Step 11 · The complete app: ask()
Every line maps to a step above: understand the question → filter → hybrid retrieve → evidence gate → answer with citations.
💡 The same question gave 45 days in March and 30 days in August. The billing clerk never saw the confidential rate; the contract manager did. The telehealth question was refused instead of invented.
Step 12 · Evaluation: measure, don’t guess
A golden set of questions with the chunk that should be found. Hit@k: was it in the top k? MRR: how high was it (1 = first, ½ = second…)? Change one thing anywhere above, re-run this cell, and compare.
Step 13 · Safety: prompt injection
The uploaded “appeal tips” file contains instructions aimed at the AI. The trusted-source filter keeps it out. Without the filter, it competes for a top spot and would reach the prompt.
🧪 Try the app
Ask anything about the library. Change the role or the date and watch the answer change. Try a question the documents cannot answer.
📄 Add your own document
Paste any text (course notes, a policy, a FAQ). Use lines starting with ## to mark sections; each section becomes a chunk. Then ask about it in the box above. Nothing leaves your browser.
Challenges
1 · Fix a paraphrase miss
The golden-set question “When does the canteen open?” fails because the documents say “cafeteria”. Add a small synonym table (canteen → cafeteria, auth → authorization, precert → precertification) and expand the question before searching. Re-run the evaluation.
2 · Tune the evidence gate
In ask(), change min_evidence to 0.3 and 0.8. Which questions start to be wrongly answered or wrongly refused? Where would you set it, and why?
3 · Better chunks
Change step 5 to index fixed-size chunks instead of heading-based ones and compare hit@1 and MRR. What does this tell you about chunking?
4 · Your own domain
Paste 5 to 10 documents from a subject you know, write 5 golden questions first, then measure and improve. For neural search, reranking and real LLM answers, continue in the Colab notebook.