What RAG evaluation misses when we only ask easy questions

The fastest way to make a document assistant look clever is to ask it a question whose answer is written plainly in the first paragraph of a file. That is a useful smoke test. It says little about what happens when two files disagree, the right file is buried, or the answer does not exist.

We have built lightweight AI interfaces and explored how they might support business workflows. We do not treat a smooth demo as proof that an assistant is ready for private knowledge. The evaluation problem is specific: did retrieval find the right evidence, and did the model answer only from that evidence? Those are separate failures with separate fixes.

Diagnose the pipeline, not only the final sentence

The RAGChecker paper proposes fine-grained measures for retrieval and generation. The authors evaluated eight RAG systems and found that component-level diagnostics reveal trade-offs a single score can hide. That is a useful framing for a product team even if it never adopts the exact framework.

Start with a small set of questions taken from real tasks. For each one, record the source that should support the answer. Add questions where the source is missing, outdated, or contradictory. Then inspect the retrieved passages before reading the final response. If the passage never arrived, a stricter prompt cannot recover it. If it arrived and the answer still wandered, changing chunk size may not solve the problem.

An honest “I cannot confirm” is a feature

Many business questions have no answer in the connected files. The assistant should say that plainly and point to a next step. Testing abstention is especially important because a polished unsupported answer is more dangerous than an obvious error. We would count an unsupported confident claim as a failure even when the sentence sounds plausible.

Permission belongs in the evaluation set too. A response can be factually correct and still be wrong to show to this user. Test the assistant with documents from different access levels and verify that citations refer to the version the user is allowed to see.

This article draws on research and product engineering judgment. It is not a claim that a specific client assistant has passed this evaluation. The practical recommendation is simple: keep a repeatable set of hard questions, read the retrieved evidence, and rerun those cases whenever the documents or model change.