What is chunking in RAG?
Chunking creates the units your RAG system searches. Each chunk can be embedded, indexed and retrieved independently. Azure's chunking guidance explains the trade-offs between context, retrieval noise and processing cost, and recommends choosing with document structure, content type, model constraints and evaluation in mind. It does not offer a universally best size, overlap or algorithm, because there isn't one.
You can return to the end-to-end RAG pipeline to see where chunking sits in the full ingestion and runtime flow.
Start with the retrieval task, not a default size
Before you choose a strategy, write down the retrieval task:
- What questions will users ask?
- What is the expected answer length?
- What document types are in the corpus?
- What is the model's context window?
- What is the acceptable latency and cost?
A setting that works for one corpus, task and model can fail on another. The Azure RAG design guide places preparation, chunking, metadata and embeddings in the ingestion stage, then recommends testing both individual stages and the system as a whole. It is a useful implementation map, not a universal blueprint.
Fixed-size, recursive, structural and semantic approaches
This strategy decision matrix is an editorial synthesis, not an official taxonomy. Use it to choose candidates for an experiment, not to pick a winner on paper.
| Strategy | Likely strengths | Failure modes | Metadata needs | Suitable experiments |
|---|---|---|---|---|
| Fixed-size | Simple, predictable and fast | Can split sentences, tables and headings; loses structure | Minimal: offset, length | Compare against a structural baseline on a representative set |
| Recursive | Can respect paragraph and sentence boundaries | May still split complex structures; sensitive to parameters | Moderate: section and paragraph boundaries | Vary recursion depth and delimiter priority |
| Document-structural | Can preserve headings, sections and tables | Struggles with unstructured or inconsistent content; produces variable sizes | High: heading hierarchy, table boundaries, section IDs | Compare on structured and unstructured subsets |
| Semantic | May group related content across surface-level boundaries | Can add computation; may over-merge or under-split; model-dependent | High: embedding similarity, topic labels | Compare against a simpler baseline on labelled retrieval cases |
Treat each row as a hypothesis to test, not a ranking. A document-structural method may suit a clean policy manual and struggle with an inconsistent export. A fixed-size baseline may be crude but valuable: if a more complex method cannot improve the agreed task measures enough to justify its operating cost, complexity has not earned its place.
Chunk size and overlap trade-offs
Chunk size affects:
- Context. Larger chunks provide more surrounding context but may include irrelevant content.
- Retrieval noise. Smaller chunks are more precise but may lack context.
- Processing cost. More chunks means more embeddings, more index entries, more retrieval candidates.
Overlap can preserve continuity across chunk boundaries. It increases index size and retrieval candidates, and may reduce the chance of splitting a relevant passage. How much it helps is something to test.
Measure the distribution of resulting chunk lengths, not just the configured maximum. A structural rule can produce a small number of unusually long sections, while aggressive overlap can make near-duplicate passages dominate the retrieved context. Inspect examples from the shortest, median and longest groups, then check whether a query at each boundary retrieves enough evidence without crowding out other relevant passages. Keep overlap, retrieval depth and context budget as separate recorded variables so that an improvement is not wrongly attributed to chunking alone.
Azure's chunking guidance recommends evaluating these trade-offs against the specific corpus and task. Treat every number as a local setting to test, not a default to inherit.
Headings, tables and mixed-format documents
Structured content brings its own boundary problems:
- Headings. A chunk that starts mid-section loses its heading context. Preserve heading hierarchy in metadata.
- Tables. A table split across chunks can lose row, column or heading context. Test keeping suitable tables intact or representing them in a structured format.
- Mixed format. A document containing prose, tables, code and images may need different rules for different content types.
Azure's chunking guidance warns that format conversion can lose information. First prepare source documents so that you can inspect what the transformation preserved, changed or dropped before chunking begins.
Metadata, provenance and surrounding context
Give each chunk enough metadata to support retrieval, permission checks and traceability:
- Source document ID and version.
- Section or heading context.
- Position within the document.
- Permission context.
- Content type (prose, table, code, image description).
One implementation pattern is to carry permissions in chunk metadata during ingestion. Metadata alone is not enforcement. The retrieval path still has to resolve and apply the user's current rights.
Build a representative chunking test set
A useful test set represents the documents and queries the system will actually face. Include:
- Documents of different types and structures.
- Queries that require single-chunk answers.
- Queries that require multi-chunk answers.
- Queries where the relevant information is at a chunk boundary.
- Queries where the relevant information is in a table or structured element.
In controlled multi-document question-answering and key-value retrieval experiments, Liu and colleagues found that performance varied with the position of relevant information. That result does not generalise to every current model or task. It does give you a good reason to include placement-sensitive cases in your own test set.
Do not build the test set only from questions that the current system already answers well. Include queries with no supported answer, ambiguous wording, relevant evidence near the start, middle and end of a long source, and cases that need two sections to be combined. Record the exact source version and the passage or passages a reviewer expects retrieval to find. Without relevance labels, a precision or recall figure may look exact while measuring the wrong target.
Repeatable chunking experiment worksheet
This worksheet is an editorial synthesis. Use a fresh copy for each controlled comparison, and decide what acceptable behaviour means before you see the candidate's results.
| Field | What to record |
|---|---|
| Retrieval task | User type, question pattern and what a useful result enables |
| Representative source set | Exact document IDs and versions, formats, structures and permission cases |
| Baseline | Current strategy, parameters and observed results |
| Candidate strategy | Algorithm, size or boundary rule, overlap, parser and metadata carried forward |
| Frozen system inputs | Embedding model, index settings, reranker, generator and prompt used across the comparison |
| Retrieval cases | Queries, labelled relevant passages, boundary cases, table cases and unsupported-answer cases |
| Answer checks | Faithfulness to retrieved context, correctness, completeness and appropriate abstention |
| Permission and provenance checks | Whether every result remains authorised and traceable to the exact source version |
| Operating measures | Index size, ingestion time, query latency and cost in the test environment |
| Error analysis | Missed evidence, noisy retrieval, broken structure, duplicates and permission failures |
| Decision | Keep, reject or run another experiment, with owner, date and rationale |
Avoid a universal pass mark. Decide the minimum acceptable behaviour for the named task before seeing the candidate's results, then record any trade-off explicitly. For example, a strategy that retrieves tables more reliably but materially increases duplicate context may need a separate table path rather than replacing the corpus-wide baseline.
Compare retrieval results, answer behaviour and operating cost
Check at least five parts of the result:
- Retrieval precision. What proportion of retrieved chunks are relevant under the test set's labels?
- Retrieval recall. What proportion of the labelled evidence needed for the task is found?
- Answer faithfulness. Does the answer follow the retrieved context?
- Answer correctness. Is the answer factually accurate?
- Latency and cost. What is the operational cost of the strategy?
RAGAS proposes reference-free measures for retrieved-context relevance, answer faithfulness and answer relevance. They are useful dimensions, not ground truth or security tests. ARES evaluates similar dimensions using synthetic training data, lightweight judges and a small human-labelled set. Neither framework makes automated judges a replacement for human review.
Azure's RAG evaluation guidance separates groundedness, completeness, context use, relevance and correctness, and recommends repeating evaluation as the system changes. Use a structured process to measure retrieval quality instead of treating one aggregate score as the answer.
For a useful comparison, freeze the corpus, queries, parser output, embedding model, retrieval depth, reranker, generation model and prompt, then change one chunking variable. Inspect results by document type rather than relying only on one average. Review the actual chunks returned for heading boundaries, tables, long sections and multi-chunk answers; a score cannot explain why evidence disappeared or irrelevant text entered the context.
Separate retrieval from answer evaluation. First ask whether the labelled evidence was available to the generator and whether unauthorised material was excluded. Then assess whether the answer used that evidence faithfully and correctly. Record latency, index size and ingestion cost alongside quality, classify failures, and repeat the same cases after the corpus, parser, embedding model or retrieval configuration changes.
Common chunking failure patterns
- Orphaned content. A chunk that contains a reference ("as shown in the table above") but not the table itself.
- Split tables. A table divided across chunks, losing its structure.
- Heading loss. A chunk that starts mid-section without its heading context.
- Over-merging. A chunk that combines unrelated content, introducing noise.
- Under-splitting. A chunk that is too large, diluting the relevant content.
- Permission gap. A chunk that loses its permission metadata during transformation.
Check source documents before transformation
Choose and test the chunking strategy, then separately check the exact source version before production transformation. These steps answer different questions.
.mdSiren is a document security workspace for AI. When it launches, Standard Scan will check the exact version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. Chunking is not sanitisation, and the document decision does not choose or validate a chunking strategy or secure the resulting RAG system. See what Standard Scan checks.
Frequently asked questions
What is the best chunk size for RAG? There is no universal best chunk size. The right size depends on your corpus, task, model and measured results. Start with a strategy that fits the document structure, then test it on representative queries.
Should RAG chunks overlap? Overlap can reduce the risk of splitting relevant content, but it increases index size and retrieval candidates. Test with and without overlap on your specific corpus and task.
Is semantic chunking always better? No. Semantic chunking can add computation and is model-dependent. It may over-merge or under-split. Compare it against structural baselines on your specific corpus.
How should tables be chunked for RAG? Consider keeping tables as single chunks or representing them in a structured format. Test whether splitting a table degrades retrieval quality for your task.
How do you test a RAG chunking strategy? Build a representative test set. Compare retrieval precision and recall, inspect the returned chunks, check answer faithfulness and correctness, and record latency and cost. Repeat the same cases when the corpus, parser, model or retrieval configuration changes.
