Build RAG one layer at a time, then learn where it breaks.
A hands-on session using one running example: a denial-management assistant for a US healthcare revenue cycle team. Every lab uses the same payer policy, so each new layer visibly fixes a failure from the one before.
Vector baseline
Embeddings, chunking, the grounded prompt, and overcoming the knowledge cutoff.
Advanced retrieval
Hybrid search, reranking, metadata filters, parent-document chunking, step-back queries.
Agentic GraphRAG
Planning, routing, self-critique, knowledge graphs and multi-hop reasoning.
- Explain RAG in plain language, and why it is not retraining
- Trace data from source document to cited answer
- Diagnose weak retrieval and pick the right fix
- Choose between standard, advanced and agentic designs
- Name the evaluation, security and observability controls for production
- Sketch a safe first version for a real use case
Why models need external knowledge
A language model answers from patterns it learned in training. It has never seen your payer contracts, and it cannot know about last week’s amendment.
The retrieval imperative 3 min
The problem
Models have knowledge cutoffs and can invent plausible details. Private data was never in training, sources are hard to verify, and you cannot fine-tune for every new fact.
The solution
RAG moves the knowledge burden out of the model’s weights into an external, updatable store that is searched at answer time.
The result
Answers grounded in supplied evidence, with citations and an honest “I don’t have enough evidence” when retrieval comes up empty.
Try it: same question, two systems
Demo“What is our current appeal deadline for Payer X authorization denials?”
The RAG idea in one sentence 3 min
RAG finds relevant evidence first, then asks the model to answer using that evidence.
“Why was claim 847 denied?”
Search denial notes and the Payer X policy.
Place the evidence into the prompt.
Compose a response with citations and limits.
Myth or fact? 1 min
Trace data from source to answer
RAG is two connected pipelines. Most failures can be located in one of them, so this map is also your first debugging tool.
Two connected pipelines 3 min
Click any stage to see what it does and how it fails
ExploreWhat it does
Typical failure
Chunking creates retrievable units 4 min
Chunking decides what the retriever can return. Small chunks improve precision; larger chunks keep context. Split the policy below and watch what breaks.
Chunking lab: Payer X, Section 4.2
LabQuality checks
Advanced pattern: parent-document retriever
201 previewSearch on small child chunks for precision, but hand the model the full parent section so it can reason. Query: “Is there an exception to the 30-day rule?”
Sent to the model
Likely answer
Embeddings support semantic search 3 min
An embedding turns text into numbers that capture aspects of meaning. Similar text lands close together, and the vector store returns the nearest neighbours.
Meaning map (2-D sketch of a high-dimensional space)
ExploreTop 3 nearest
What to notice
The grounded prompt 3 min
A grounded prompt has three inputs: behaviour instructions, retrieved evidence and the question. Change each one and watch the answer shift.
Prompt builder
LabModel answer
Grounding needs both relevant evidence and clear answer rules. Irrelevant context makes the system confidently grounded in the wrong material.
Walkthrough: a denial policy assistant 2 min
“Why was the claim denied and what should we attach?”
Step throughWhy was the claim denied and what should we attach?
authorization · CARC CO-197 · Payer X · service date 2026-08-12
Denial note + Payer X policy §4.2 v3 + claim metadata
Cause, appeal deadline, attachments, each with a citation
Improve relevance before you touch the model
The 101 baseline depends entirely on semantic similarity: if the answer does not look like the question in vector space, retrieval fails. Four layers fix most of that.
Hybrid retrieval: dense + sparse 4 min
Semantic (dense)
Finds meaning. Wins when wording differs: “prior authorization” vs “precertification”.
Keyword (sparse, BM25-style)
Finds exact terms. Wins on codes, IDs, drug names and clause numbers: CO-197, CPT 99213, §4.2.
Hybrid search lab: 8-document mini corpus
LabSemantic only
Keyword only
Hybrid
Green rows are the documents a domain expert marked relevant. Semantic scores are simulated; keyword scores are computed live from term overlap. Verdict = relevant document in the top 3.
Reranking improves the shortlist 3 min
Fast retrieval gathers candidates for recall. A reranker spends more compute ordering the best few for precision.
Rerank the candidates
DemoReranking cannot recover a document that never entered the candidate set. If recall is poor, fix indexing, query rewriting, filters or candidate count first.
Metadata, freshness and access control 3 min
Metadata turns similarity search into business-aware retrieval. Filters run on structured fields before anything reaches the model.
Filter lab: what reaches the model?
LabRisk check on what the model receives
Query transformation: step-back prompting 2 min
Use a model to rewrite or abstract the user’s question before retrieval. A step-back query asks the broader question, so foundational context is found even when the specific wording matches nothing.
Rewrite, then retrieve
DemoOriginal query
Why did claim 847 get CO-197 on 12 Aug when we got verbal approval from the payer rep?
Retrieves
- 1Call-log note on claim 8470.71
- 2Generic appeal letter guide0.52
Step-back query
(press “Step back”)
Adds
Agentic RAG and knowledge graphs
Standard RAG is a single pass: retrieve once, answer once. It cannot notice missing evidence, change strategy, or connect facts spread across documents.
The one-shot limitation 1 min
The bottleneck
A static, single-pass pipeline.
Failure modes
Cannot self-correct or adjust its search. Fails at multi-hop reasoning across disconnected documents.
The verdict
Fine for single-hop fact lookups; fragile for analytical questions like “compare Q3 to Q1 and explain why”.
The agentic loop: plan, route, act, critique 3 min
Watch an agent answer a multi-part question
SimulationQuestion: Compare Q3 vs Q1 CO-197 denial rates for Payer X, and tell me whether the July policy change explains the difference.
Figures in the simulation are illustrative.
The knowledge-graph advantage 3 min
Vector stores treat chunks as isolated points. Knowledge graphs store explicit relationships between entities, so the connection between two distant documents can itself be the answer.
“Which open claims are affected by the July amendment?”
ExploreGraphRAG and Text2Cypher 2 min
Indexing time
An LLM extracts entities and relationships, builds the graph, then detects and summarises communities of closely related data.
Query time
Local search: deep dives on specific entities and their neighbours. Global search: map-reduce over community summaries for broad, thematic questions.
Text2Cypher: natural language to graph query
DemoGuardrails: read-only credentials, a schema allowlist, query validation before execution, and row limits.
The evolution matrix 1 min
| Architecture | Query complexity | Retrieval | Flow | Best for |
|---|---|---|---|---|
| Standard RAG | Simple | Vector only | Linear, one shot | Single-hop fact lookup |
| Advanced RAG | Nuanced | Hybrid + smart chunking + rerank | Linear, optimised | Deep document Q&A, precise recall |
| Agentic GraphRAG | Complex, multi-part | Multi-tool + knowledge graph | Cyclical, iterative | Synthesis, multi-hop reasoning, research |
Pick the architecture
ExerciseArchitectural rule: use static RAG for single-hop facts. Build agentic GraphRAG only for cross-source, multi-step reasoning, where its extra cost pays for itself.
Evaluate, secure and operate
A good answer can hide weak retrieval, and poor wording can hide excellent retrieval. Measure each layer separately and design for failure.
Evaluation separates retrieval from generation 4 min
Retrieval
Did we find the needed evidence?
Recall@k · Precision@k · MRR
Groundedness
Does every claim follow from context?
Faithfulness · claim support
Answer quality
Did it solve the user’s task?
Correct · complete · useful
Operations
Can it meet its constraints?
Latency · cost · failure rate
Retrieval metrics calculator
LabGolden question: “What is the CO-197 appeal deadline for Payer X, and what must we attach?” The expert marked 3 documents as relevant. Tick the results you judge relevant and move k.
Security and failure controls 3 min
RAG opens a new attack surface: external text enters the prompt. Retrieved content can carry instructions or sensitive data.
Spot the prompt injection
ExerciseThese three chunks were retrieved for “What should we attach to a CO-197 appeal?” Which one should never be followed as an instruction?
Prompt injection Treat retrieved text as data, not instructions
Delimit retrieved content, strip or flag imperative patterns, use source allowlists, and never let retrieved text change tools, permissions or output destinations.
Sensitive data Minimise, mask, encrypt, audit
Keep PHI exposure to the minimum necessary. Mask identifiers at ingestion where possible, encrypt at rest and in transit, and log access.
Unauthorised retrieval Filter by identity and role before generation
Access control is enforced in the retrieval layer using the user’s identity, not by asking the model to “not reveal” things.
Unsupported answer Require citations; allow “insufficient evidence”
Check that each cited chunk exists in the retrieved set, and make refusal an acceptable outcome in evaluation.
Index drift Monitor ingestion, freshness and deletion
Alert on ingestion lag and failed updates; verify deleted or superseded documents are gone or correctly versioned.
Production architecture and observability 3 min
You must be able to reproduce a bad answer. Trace every request across components, and version everything that can change an answer.
Request trace · click a span
ExploreSpan detail
Log per request
Query, filters applied, retrieved chunk IDs and scores, prompt version, model, index version, latency by component, user feedback.
Operational questions
Can we reproduce a bad answer? Did the source change? Which component added latency? Did access filters apply?
Design the first safe version
A contract manager asks: “Can we appeal this authorization denial, and what evidence is required?” Pairs take 60 seconds, then debrief for 60.
Applied challenge
PairsAnswer key
A wrong answer can start in data, retrieval, prompting or generation. Diagnose the layer before tuning. A stronger model cannot reliably compensate for missing, unauthorised, stale or irrelevant evidence.
A practical RAG checklist
Next step: build a 20-question evaluation set before choosing advanced components.