Retrieval-augmented generation (RAG) is often described as “search, then ask a model.” That description hides the parts most likely to break: document parsing, chunk boundaries, access rules, retrieval, source selection, and the model’s decision to answer or abstain.
If a result is wrong, changing the prompt may not help. The system may have retrieved the wrong paragraph, missed a newer policy, or included a document the user should not see. Treat RAG as a pipeline whose stages can be inspected and tested independently.
Build a small evaluation set first
Collect real questions that represent the product’s intended use. For each one, record the expected source documents and the facts a good answer must contain. Include questions with no answer in the corpus, ambiguous wording, outdated material, and users with different access levels.
This does not need to be a huge dataset to be useful. A carefully reviewed starter set gives the team a repeatable baseline and exposes important failure classes before a launch. Keep sensitive data out of test fixtures unless there is a clear need and appropriate access controls.
Test ingestion before retrieval
Documents need stable identifiers, source metadata, update timestamps, and any access-control labels required by the product. Confirm that text extraction preserves meaningful structure such as headings, tables, and lists. If a file changes, the index should not quietly retain stale chunks as if they were current.
Chunking is a trade-off. Very small chunks can lose the context that gives a sentence meaning; very large chunks may dilute retrieval and consume the context window. Start with boundaries that follow the source structure, then evaluate whether the retrieved passages contain enough context to answer the questions.
Measure retrieval separately
For each test question, inspect whether the expected source appears among the retrieved results. This is retrieval quality, not answer quality. A model cannot reliably cite a passage it never received.
Review the full retrieval trace: query transformation, filters, result order, scores, and the content sent downstream. Test access filters with users who should and should not see particular records. Authorization must be enforced in the retrieval layer; asking the model to ignore unauthorized text is not access control.
Evaluate the answer and its evidence
A useful answer should be supported by the retrieved sources, distinguish facts from inference, and say when the available evidence is insufficient. Require citations that point to the source passages or records actually supplied to the model—not merely to a broad collection.
Score answers against a rubric that reflects the product: factual support, completeness, correct source attribution, and appropriate abstention. Human review remains important, especially when the response can influence financial, legal, health, or operational decisions.
Improve one stage at a time
When a test fails, label the failure before changing the system:
- Ingestion: the needed information is missing or malformed.
- Retrieval: the relevant passage is not found or ranked well.
- Context: the right source is present but key surrounding information is absent.
- Generation: the answer misreads evidence, invents details, or omits uncertainty.
- Authorization: the system returned information outside the user’s permissions.
Then change one stage and rerun the same evaluation set. Compare results against the previous baseline so an improvement in one area does not conceal regressions elsewhere.
Keep the corpus trustworthy
Define who can publish, update, and remove indexed material. Record where each chunk came from, when it was last refreshed, and which access policy applies. Provide a way to remove content from the index when its source is deleted or its permissions change.
RAG quality is not a single model score. It is the combined behavior of data, retrieval, permissions, and generation. Make those boundaries visible, and the system becomes easier to debug, safer to operate, and more honest about what it knows.
