How to Test a RAG Chatbot
By TBS Tech Services QA team. Published
A RAG chatbot has two parts to test: the search that finds passages in your documents, and the model that writes an answer from them. Test them separately so you know which to fix. This guide covers golden sets, retrieval checks, groundedness, citations, access control and the regression run after each change.
This guide is a working outline. We will expand it with worked examples.
Why test retrieval and generation separately?
If a RAG chatbot gives a wrong answer, the cause is either that the search didn’t find the right passage, or that the model ignored or stretched what it found. These need different fixes. Testing each part on its own tells you where to look instead of guessing.
How do you build a golden set for RAG?
Collect real questions, write the expected answer for each, and mark which passages in your documents support it. Have the document owner approve every entry. Include questions with no answer in the documents, questions that look alike but differ, and questions each user role should or shouldn’t be able to ask.
How do you test retrieval?
For each golden question, run only the search step and check whether the supporting passage appears in the results, and how high. Track how often it’s found at all, and how often it’s found near the top. A retrieval miss means the model never had a chance to answer correctly.
What usually causes retrieval misses?
Most misses come from how documents are prepared, not from the model. Pieces that are too large or too small, tables and scanned PDFs that parse badly, a stale index and questions worded differently from the documents are the usual suspects. Fixing these often helps more than changing the model.
How do you test the answer?
Give the model the retrieved passages and check three things. Is every claim in the answer supported by those passages? Does the answer actually address the question? Do the citations point to the passage that supports each claim? Use code for facts and format, and a calibrated judge only for fuzzy qualities.
What happens when the answer isn’t in the documents?
The chatbot should say so plainly and, where possible, point to where help is available. Test questions that are close to your content but not covered, questions about topics you removed, and questions that contradict your documents. A confident answer to any of these is a failure.
How do you test access control?
Sign in as users with different permissions and ask for content only another role should see. Check that neither the answer nor the cited sources reveal it. Test the filter itself, not only the chatbot, because a retrieval layer that ignores permissions leaks data even when the answer looks harmless.
What should you re-test after a change?
Re-run the full golden set after any change to chunking, embeddings, the search method, the prompt, the model or the documents. Compare retrieval and groundedness against the last run. Small changes can quietly lower quality, and a regression suite is the dependable way to notice.
See how we apply this in RAG evaluation, and read the methodology for how we judge answers.