Building RAG Systems That Don't HallucinateBuilding RAG Systems That Don't Hallucinate

The first RAG demo I built answered five questions perfectly and then confidently invented a policy that did not exist. My instinct was to blame the model and rewrite the prompt.

The prompt was fine. The retriever had returned three chunks about a completely different document, and the model did what models do: it worked with what it was given.

Most hallucinations in RAG are retrieval failures. Fix retrieval first.

Chunking is the highest-leverage decision#

Chunking determines what can ever be retrieved. Get it wrong and no amount of prompt engineering recovers.

Fixed-size chunks split mid-sentence. A 512-token window that ends halfway through a definition produces an embedding for half an idea. It'll match nothing cleanly.

Structure-aware chunking works far better. Split on document structure, then merge small pieces up to a target size:

from langchain_text_splitters import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
)

The separator order matters: try to break on headings, then paragraphs, then sentences, and only fall back to characters. The overlap keeps a sentence that straddles a boundary retrievable from either side.

Then add context back. A chunk from the middle of a document has lost the title, the section, and the source. Prepend it before embedding:

def contextualize(chunk: str, doc_title: str, section: str) -> str:
    return f"Document: {doc_title}\nSection: {section}\n\n{chunk}"

This one change produced the biggest single quality jump in the pipeline I built. Chunks that read as "…must be filed within 30 days." become "Document: Claims Policy / Section: Deadlines — …must be filed within 30 days." Suddenly a query about claim deadlines actually matches.

Vectors alone are not enough#

Embeddings capture meaning and are bad at exact tokens. Search for an error code, a SKU or a person's name and dense retrieval will cheerfully return something semantically adjacent and factually wrong.

Run both, and fuse:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    """RRF — combines rankings without needing comparable scores."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)
 
dense  = vector_store.similarity_search(query, k=20)
sparse = bm25_index.search(query, k=20)
fused  = reciprocal_rank_fusion([[d.id for d in dense], [d.id for d in sparse]])

RRF is the right default because it only needs ranks, not scores — so you don't have to normalise a cosine similarity against a BM25 score, which is an apples-to-oranges problem people waste a lot of time on.

Retrieve wide, then rerank narrow#

Bi-encoders (what your vector store uses) embed the query and documents separately. That's fast and approximate. A cross-encoder reads query and document together and is dramatically more accurate — and far too slow to run over a whole corpus.

So: retrieve 20–50 candidates cheaply, rerank them properly, keep the top 5.

from sentence_transformers import CrossEncoder
 
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
pairs  = [(query, doc.text) for doc in candidates]
scores = reranker.predict(pairs)
 
top = [doc for doc, _ in sorted(zip(candidates, scores), key=lambda x: -x[1])[:5]]

Then apply a relevance floor. If the best reranked score is below threshold, the corpus doesn't contain the answer:

if not scores or max(scores) < RELEVANCE_FLOOR:
    return "I don't have information about that in the available documents."

Answering "I don't know" is a feature. A RAG system that always produces an answer is a system that will confidently produce a wrong one.

Ground the generation#

Two things do most of the work: forcing the model to work only from the context, and requiring citations.

SYSTEM = """You answer strictly from the provided context.
 
Rules:
- Use only information present in the context below.
- Cite the source id in brackets after each claim, e.g. [doc_3].
- If the context does not contain the answer, say exactly:
  "I don't have information about that in the available documents."
- Never use prior knowledge to fill gaps.
- If sources conflict, say so and cite both."""
 
context = "\n\n".join(f"[{doc.id}] {doc.text}" for doc in top)

Citations aren't only for the user. They're a verification hook — you can programmatically check that every cited id was actually in the context, and flag any answer that cites something you never sent:

cited = set(re.findall(r"\[(\w+)\]", answer))
provided = {doc.id for doc in top}
 
if not cited <= provided:
    log.warning("Model cited sources not in context: %s", cited - provided)

That check catches a specific and nasty failure mode: the model inventing a plausible-looking source id to satisfy the format.

Evaluate the stages separately#

This is what separates a demo from something you can improve. If you only measure end-to-end answer quality, you can't tell whether a bad answer came from bad retrieval or bad generation — so you tune blindly.

Measure them apart:

Retrieval — did the right chunk come back at all?

Generation — given good context, was the answer faithful?

Build a golden set of 50–100 real questions with known correct sources. It's a tedious afternoon and it's the difference between engineering and vibes.

def recall_at_k(golden: list[Case], retrieve, k: int = 5) -> float:
    hits = sum(
        1 for case in golden
        if any(doc.id in case.relevant_ids for doc in retrieve(case.query, k=k))
    )
    return hits / len(golden)

If recall@5 is 0.6, no prompt will save you — 40% of the time the answer isn't in the context at all. Fix chunking and retrieval before touching the prompt.

The failure modes, ranked#

From what I've actually debugged:

1. Chunks too small or split mid-idea — the answer exists but is fragmented across chunks. Fix chunking.
2. Missing context in the chunk — the text is right but doesn't mention what it's about. Prepend title and section.
3. Semantic search missing exact terms — add BM25, fuse with RRF.
4. Top-k too low — the answer was at rank 8 and you kept 5. Retrieve wide, rerank.
5. No relevance floor — the system answers from irrelevant context. Add the threshold.
6. Prompt permits prior knowledge — tighten it and require citations.

Notice that five of the six are retrieval, not generation.

The pipeline, end to end#

query
  → embed + BM25 (retrieve 40 candidates)
  → RRF fusion
  → cross-encoder rerank → top 5
  → relevance floor check → bail out if too low
  → grounded prompt with cited context
  → verify citations against provided ids
  → answer + sources

Every stage is independently measurable, and that's the actual point. When quality drops after a corpus update, you want to know which stage regressed — not to start rewriting prompts and hoping.