RAG evaluation: metrics, test sets and release gates

What must a RAG evaluation answer?

Start with the decision, not the metric. A team usually needs to know whether a system can retrieve the right evidence for a defined task, place useful context in front of the model, produce an acceptable answer, respect access boundaries and behave predictably when something changes.

That means evaluating several layers:

• Ingestion: did the expected source content and metadata survive preparation?
• Retrieval: did the system find useful evidence and avoid distracting material?
• Context: did the selected passages cover what the answer needed?
• Answer: was the response faithful, correct, complete and well attributed?
• Permissions: did the system exclude material the current user could not access?
• Security and failure behaviour: did harmless test inputs produce the defined safe outcome?
• Operational outcome: did the system help with the real user task at an acceptable cost and speed?

These questions belong to one evaluation programme, but they do not share a universal score. A polished answer can hide weak retrieval. A faithful answer can repeat an error in the supplied context. A permission failure can occur even when every quality metric looks strong.

Before writing tests, map the RAG pipeline under test. The boundary should include the ingestion and runtime components that can change a result, not just the final model call. The Azure RAG design guide supports evaluating ingestion, runtime stages and the complete system with representative content and queries. Its Azure-oriented topology is a useful decomposition, not a universal architecture.

Start with user tasks and representative test sets

A RAG test set should be a small, maintained model of the work the system is expected to do. Begin with real task categories rather than a convenient list of trivia questions. A policy assistant might need to locate a clause, compare two current procedures, explain an exception, decline when evidence is missing and cite the exact source version used.

For each test case, record:

• the user and task being represented;
• the question or request in realistic language;
• the source version and evidence expected to support an answer;
• acceptable variations in the answer;
• an explicit unanswerable outcome where the corpus lacks support;
• the permission state under which the case should run;
• the consequence of a wrong, incomplete or over-confident result; and
• who approved the case and when it should be reviewed.

Coverage matters more than raw test-set size. Include common tasks, important rare tasks, ambiguous wording, missing evidence, conflicting sources, recently changed documents and queries from different user groups. Keep a stable regression set, then add a rotating sample from recent real use where collection and review are permitted. Otherwise the test set can become an exam the system has effectively been tuned to pass.

Expected evidence is often more useful than one rigid reference answer. It lets reviewers distinguish a response that reaches the right conclusion in different words from one that sounds right without using the required source. Where several answers could be valid, record the decision criteria and examples of acceptable boundaries rather than pretending there is one canonical sentence.

Measure retrieval behaviour

Retrieval evaluation asks what entered the candidate set, where the expected evidence appeared and what irrelevant material came with it. Inspect individual cases before averaging them. A reasonable summary score can conceal a complete miss on the highest-consequence task.

Useful retrieval observations include:

• whether the expected source or passage appeared in the retrieved set;
• its position before and after reranking;
• whether the retrieved passage contains enough surrounding meaning;
• which distracting passages ranked above it;
• whether source, version and permission metadata remained attached; and
• whether an unanswerable query returned no adequate evidence rather than a plausible near-match.

Teams can turn these observations into workload-specific rates or ranking measures, but the unit and cut-off must be explicit. “Relevant at the document level” and “the exact supporting passage appears within the context budget” answer different questions.

Chunking is one variable in this layer. Microsoft’s RAG chunking guidance describes trade-offs among context, retrieval noise and processing cost, with choices shaped by document structure, content and model constraints. It does not provide a best chunk size. When retrieval misses or returns fragments without enough meaning, test alternative chunking strategies as a controlled experiment while holding the rest of the pipeline as steady as practical.

Record the configuration used for every run, including corpus snapshot, parser, chunking policy, embedding and retrieval components, filters, reranker and retrieval count where relevant. These fields are comparison coordinates, not a recommendation for a particular stack.

Evaluate context relevance and coverage

Retrieved candidates are not always identical to the passages ultimately supplied to the model. Context assembly may remove duplicates, apply filters, reorder passages, trim content or combine results from several retrievers. Evaluate the assembled context as its own artefact.

Ask two different questions:

1. Relevance: how much of the supplied context helps answer this test case?
2. Coverage: does the context contain all material evidence needed for an acceptable answer?

High relevance with poor coverage can produce a concise but incomplete answer. High coverage with heavy noise can make the model’s job harder and consume the context budget. Inspect the missing and distracting passages, not only the final rating.

RAGAS proposes reference-free measures aimed at retrieved-context relevance, answer faithfulness and answer relevance or quality. The RAGAS paper is useful for separating those dimensions, but its measures are not ground truth and do not show that retrieved context is true, current or authorised.

Ordering can be worth testing as another system variable. In controlled multi-document question-answering and key-value retrieval experiments, the Lost in the Middle study found that performance varied with the placement of relevant information and was often lower when it appeared in the middle of long inputs. That does not establish one ordering rule for every current model or workload. It supports including placement-sensitive cases when the production system assembles long contexts.

Separate faithfulness, correctness, completeness and citation quality

Answer evaluation becomes clearer when reviewers score distinct properties rather than “good” or “bad”. The Azure guidance on the RAG evaluation phase treats groundedness, completeness, context use, relevance and correctness as separate, workload-dependent dimensions.

• Faithfulness or groundedness: are the answer’s material statements supported by the context that was supplied?
• Correctness: are those statements accurate against the accepted source of truth?
• Completeness: does the answer cover the parts needed for this user task?
• Citation quality: do citations point to the source passage and version that actually support each material claim?
• Answer relevance: does the response address the request without avoidable detours?

These dimensions catch different failures. A faithful answer can be wrong because the retrieved document is outdated. A correct statement can be unsupported by the retrieved context. A complete-looking answer can blend sources without showing which source supports which claim. A valid-looking citation can point to a passage that does not support the sentence beside it.

Build the rubric around the intended task. For a low-consequence orientation tool, a concise answer with clearly signposted uncertainty may be acceptable. For a decision with material consequences, the gate may require direct source checking or a human reviewer. Those are local policy choices. No passing score certifies factual accuracy, authorisation or safety.

Combine human and model-based review carefully

Human review is valuable when judgement, domain nuance or consequence matters. Model-based review can help apply a clear rubric across a larger test set, find candidates for inspection and speed up repeated comparisons. Both can be inconsistent, and neither should be treated as an objective oracle.

ARES is one worked example. The ARES paper evaluates context relevance, answer faithfulness and answer relevance using synthetic training data, lightweight judges and a small human-labelled set across its reported question-answering tasks. Its results do not transfer automatically to another domain, corpus, model or threat model, and the framework does not show that automated judges replace human review.

A practical review design can include:

• a written rubric with examples and clear failure boundaries;
• human-labelled cases chosen from each important task and risk slice;
• judge prompts and versions recorded with the system under test;
• periodic comparison between judge and human decisions;
• review of disagreements rather than automatic majority voting;
• blinded comparison where reviewers should not know which system produced an answer; and
• escalation rules for uncertain or high-consequence cases.

Avoid using the same model configuration as generator, sole judge and final arbiter without testing the resulting bias. A judge score is evidence produced by a method. It is not the underlying truth. When the judge, rubric or model changes, treat that as an evaluation change and rebuild the baseline where needed.

Add permission and adversarial test cases

Quality tests usually assume the right material is available. Permission tests ask whether it is available to the right person under the current policy. Include a permitted user who should retrieve a source, a user who should not, a recently revoked user, a newly granted user and a case involving cached or previously generated output where that path exists.

Microsoft’s document-level access-control example for Azure AI Search shows one implementation that projects permission metadata to indexed chunks and applies caller identity or permissions during retrieval. Several native mechanisms use the 2026-08-01-preview API. This is a named implementation, not proof that service-level roles or any index automatically preserve source permissions. Teams still need to include access-control test cases that verify permission metadata is present, current and enforced in every relevant path.

Security evaluation needs its own cases and evidence. Research has demonstrated defined conditions in which retrieved adversarial instructions changed model or application behaviour. The More than you’ve asked for paper supports testing that mechanism, but it does not establish prevalence, applicability to every current architecture or a consequential real-world outcome.

Knowledge poisoning is another bounded test area. PoisonedRAG reports attacker-chosen answer effects under specific corpus, insertion, target-question, model and evaluation conditions. Those experiments do not establish universal exploitability or real-world harm. Use harmless synthetic records in an isolated test corpus, and define the expected retrieval and answer behaviour before running the case.

Benchmarks can help organise repeatable tests. BIPIA provides tasks that test whether instructions embedded in external content cause defined output deviations, plus defence evaluations within that benchmark. Its results are not prevalence estimates and cannot be assumed to transfer unchanged to a production system.

Keep the evidence ladder intact. First, an instruction-like signal may occur in a document. Second, a defined test may show a model or pipeline behaviour change after processing it. Third, a consequential outcome may occur. Evidence for one step does not establish the next, and a finding does not prove malicious intent. Include system security tests in a separate track with harmless inputs, constrained actions and explicit expected outcomes.

The NCSC’s prompt-injection guidance recommends reducing likelihood and impact with system-level measures such as constrained privileged actions, deterministic controls and monitoring. It is expert guidance, not a security score or proof that a particular input is malicious.

Set baselines and workload-specific thresholds

A baseline is a reproducible result for a named system version, corpus snapshot and test set. It gives the team a point of comparison. It should not quietly become an eternal target after the workload, user population or consequences change.

Set thresholds only after deciding what failure means. A high-consequence permission test may be a hard gate. A modest retrieval improvement on a low-impact task may be a trade-off against latency or cost. Some dimensions need a per-case rule; others can use a distribution or slice-level summary. Always keep serious failures visible even when an aggregate improves.

Document four things for each threshold:

1. the metric or observable outcome;
2. the task slice and system version to which it applies;
3. the reason the boundary is acceptable for this workload; and
4. the person authorised to accept an exception.

Do not import a threshold from a paper, vendor dashboard or another team without showing that its task, corpus and consequences match. The threshold is a local decision informed by evidence. It is not a property of “RAG” in general.

Copy-and-adapt RAG evaluation scorecard

The scorecard below is an editorial synthesis, not a standard or universal scoring model. Copy it, adapt the fields to the workload and keep the underlying case-level evidence.

User task	Test case	Source or expected evidence	Retrieval observation	Answer dimension	Permission or security check	Reviewer or method	Workload-specific threshold	Result	Release decision	Follow-up owner
Locate the current procedure	Realistic query plus a defined missing-evidence variant	Approved source, exact version and supporting passage	Expected passage position, noise and metadata	Faithfulness, correctness, completeness and citation quality recorded separately	Permitted and denied user cases; harmless failure case where relevant	Named human reviewer, recorded judge version or both	Locally approved per-case or slice rule	Pass, fail or review with evidence link	Release, hold, restrict or rerun	Named owner and due action

Add rows for every material task and user group. A release summary may roll up results, but it should link back to the cases, configurations, reviewer decisions and unresolved failures from which it was built.

Turn results into release gates

A release gate turns evaluation evidence into an accountable decision. It names the system version, scope, required checks, blocking failures, accepted limitations, approver and follow-up date. “The average score increased” is not enough.

A practical gate might require:

• the stable regression set to run against the release candidate;
• no unresolved failure on a defined critical permission or task case;
• retrieval and answer results to meet their workload-specific thresholds;
• material regressions to have an owner and disposition;
• the configuration, corpus and source versions to be reproducible; and
• known limitations to be visible to the people operating and using the system.

The release decision applies only to the evaluated version and scope. It does not approve the wider organisational use of AI, supplier terms, data handling or a new business process. Teams should assess the wider AI use case through the relevant governance process, with accountable owners for questions outside system performance.

Keep the options more precise than pass or fail. A team may release to a narrower user group, hold the change, remove a failing source, disable a feature, require human review for a task slice or rerun after a fix. Record the reason and owner. This is a locally designed release method, not an industry standard.

Monitor regressions as the corpus and system change

Evaluation is not finished at launch. Documents change, permissions are revoked, queries shift and system components are updated. The Azure evaluation guidance recommends repeating evaluation as the corpus, queries and system change.

Define triggers for a full or targeted rerun. Useful triggers include:

• adding or replacing a material source collection;
• changing parsing, chunking, embedding, indexing, filters or reranking;
• changing the model, system instructions, context assembly or judge;
• changing identity, permission projection, cache or policy behaviour;
• discovering a production failure or a new high-value user task; and
• applying a versioned security or operational update.

The NCSC secure operation and maintenance guidance supports monitoring and evaluating versioned changes because updates can alter behaviour. It is high-level lifecycle guidance, not evidence that a particular control works.

Store enough evaluation history to compare like with like: release version, corpus snapshot, test-set version, configuration, reviewer or judge version, results, decision and owner. When a test changes, record whether it was corrected, made harder or added to cover a newly observed task. Otherwise an improving score may simply reflect an easier test set.

Monitor production signals separately from offline scores. Useful signals can include unanswered tasks, retrieval misses confirmed by review, permission denials, user corrections and cases sent for human handling. Define collection and retention according to the sensitivity of the system. Production feedback should improve the test set, but it should not silently overwrite the stable regression set.

Keep source-document checks separate from end-to-end evaluation

Document checks and RAG evaluation answer different questions. A source-level check can report defined signals in one document version. End-to-end evaluation tests what the retrieval and generation system does with a corpus, users, permissions and a particular configuration. One cannot stand in for the other.

After source approval, classification, minimisation and permission decisions, an exact-version document checkpoint can sit before ingestion. .mdSiren is a document security workspace for AI. When it launches, Standard Scan will check the exact version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. This document decision does not measure retrieval quality, answer quality, permissions or end-to-end security; source checks and system evaluation remain separate. See what Standard Scan checks.

Keep both records tied to exact versions. If a document changes, rerun the relevant source-level process and the targeted system tests affected by that change. A source result does not authorise ingestion, and a passing RAG evaluation does not create a universal safety verdict.

Frequently asked questions

How do you evaluate a RAG system?

Define the user tasks and consequences, build representative cases with expected evidence, inspect retrieval and assembled context, score distinct answer dimensions, run permission and security cases, then apply workload-specific release gates. Record the corpus, configuration, reviewers, results and owner so the evaluation can be repeated.

Which RAG evaluation metrics should be used?

Choose measures that answer the release question. Common dimensions include retrieval relevance and coverage, context relevance, answer faithfulness, correctness, completeness, citation quality and task outcome. Permission and security behaviour need separate tests. No single metric or aggregate score covers all of them.

What is the difference between faithfulness and correctness?

Faithfulness asks whether an answer is supported by the context supplied to the model. Correctness asks whether the answer is accurate against the accepted source of truth. A response can be faithful to an outdated or incorrect passage, so the two must be reviewed separately.

Can an LLM judge replace human evaluation?

No general result supports that conclusion. A model-based judge can apply a rubric at scale and help prioritise review, but its decisions depend on the model, prompt, examples and task. Calibrate it against human-labelled cases, inspect disagreements and keep people responsible for consequential decisions.

How often should RAG evaluation be repeated?

Run a defined suite before each material release and rerun targeted tests when the corpus, permissions, retrieval stack, model, prompts, context assembly, judge or user tasks change. Keep a stable regression set so changes remain comparable over time.