How to prepare documents for RAG

Use this pre-ingestion checklist and manifest to make decisions, transformations and exceptions visible before a document enters the pipeline.

What does it mean to prepare documents for RAG?

Preparation is where an eligible source document becomes an inspectable, versioned and permission-aware input for the ingestion process. It sits between source approval and chunking.

The Azure RAG solution design guide treats document preparation, metadata, embeddings and persistence as ingestion concerns. In this guide, preparation produces the structured, versioned and metadata-enriched representation that chunking receives. The Azure topology is a useful way to separate the stages, not a universal design.

For the wider process, see where preparation sits in the RAG pipeline. That guide maps the ingestion and runtime flow around this hand-off.

The following RAG document-readiness checklist is an editorial synthesis. Adapt it to your organisation's sources, policies and constraints.

Check	Question	Owner	Evidence
Use-case approval	Is this document approved for the specific use case?	Data owner / policy	Signed-off approval record
Source owner	Who owns this document and is responsible for its accuracy?	Data owner	Named owner, contact
Source location	Where does the document live, and how is it accessed?	Engineering lead	System name, access method
Sensitivity / classification	What is the sensitivity classification of this document?	Classification owner	Classification label, date
Intended users	Who is this document intended for?	Data owner	User group or role list
Permission source of truth	Where are the document's permissions defined and maintained?	Security lead	Permission system, sync mechanism
Exact version or hash	What is the exact version or content hash of this document?	Engineering lead	Version number, hash, timestamp
Parser and configuration	Which parser will process this document, and with what configuration?	Engineering lead	Parser name, version, config
Conversion-loss review	Has the parsed output been inspected for information loss?	Data owner	Inspection record, exceptions
Metadata	What metadata (provenance, permissions, lifecycle) is attached?	Engineering lead	Metadata schema, values
Update / delete path	How will this document be updated or deleted?	Engineering lead	Refresh mechanism, deletion path
Exception owner	Who handles exceptions (parse failures, permission conflicts)?	Data owner	Named exception owner
Acceptance evidence	What evidence shows this document is ready for ingestion?	Evaluation owner	Acceptance test results

Copy this source-document manifest into the work item for each exact version. It is an editorial template, not a universal metadata schema:

Keep the manifest beside the preparation evidence rather than relying on fields held only in a vector index. Update it when the source, parser, configuration, classification decision or intended use changes. An empty field is an unresolved decision, not permission to assume a default.

Decide which sources are eligible before processing

Eligibility starts with an accountable decision, not a technical transformation. A document is eligible for a specific use case only when:

• The use case has been approved by a named owner.
• The document's content is appropriate for the task.
• The document's sensitivity classification is compatible with the use case.
• The document's permissions can be preserved through retrieval.
• The source system is approved and accessible.

A document can be topically perfect and still be ineligible. If it is restricted to another team, relevance does not grant access. Make the eligibility decision before parsing or transformation begins.

Record ownership, provenance, permissions and exact version

Record at least four things for every document that enters the pipeline:

• Owner. The person or team responsible for the document's accuracy and currency.
• Provenance. Where the document came from, who created it, and when it was last modified.
• Permissions. The access rules that apply to the document, and where those rules are maintained.
• Exact version. The specific version or content hash of the document being processed.

The Azure document-level access control documentation illustrates the need to project relevant permission metadata to indexed chunks. That implementation is Azure-specific and partly in preview; metadata can be absent, stale or incorrectly applied.

Treat a changed document as a new version. If the system processes version 3 of a policy document and version 4 is published, the update path should detect the change, reprocess the document and replace or supersede the old indexed representation.

Parse and inspect conversion output

Parsing converts a raw document into a structured representation. Choose the parser according to the format and the information the use case needs to preserve.

The Azure RAG chunking phase guide supports the idea that keeping loading and chunking separable makes intermediate output easier to inspect. Format conversion can lose information. A successful parse is not proof that the source is authorised, current, complete or safe.

Do not stop at a successful parse message. Inspect the output itself for:

• Information loss. Are tables, headers, footers, images or other structural elements preserved or lost?
• Encoding issues. Are special characters, diacritics or non-Latin scripts handled correctly?
• Ordering. Is the document's logical order preserved?
• Completeness. Is the entire document parsed, or are sections missing?

Record any conversion losses as exceptions. An exception is a reason to review, not proof of a problem.

Normalise content without silently removing meaning

Normalisation makes content more consistent for retrieval and generation. It may include:

• Standardising whitespace and line breaks.
• Removing or preserving headers and footers.
• Converting tables to a text representation.
• Standardising date and number formats.

Do not make the text tidier by silently changing its meaning. If a table becomes text, check that the relationships between its cells remain understandable. If a header is removed, record the change.

The useful result is a representation that remains faithful to the source while being workable for chunking and retrieval.

Add metadata for retrieval, permissions and lifecycle operations

Metadata carries the references and attributes needed to interpret content in the index. It cannot preserve or enforce permissions by itself. Subject identity, current source policy and retrieval controls still have to meet at runtime. Depending on the architecture, each chunk may need:

• Provenance metadata. Source document ID, version, owner, creation and modification dates.
• Permission metadata. The access rules that apply to the source document, projected to the chunk.
• Lifecycle metadata. Ingestion date, last refresh date, deletion flag, superseded-by reference.
• Classification metadata. The sensitivity classification of the source document.

The Azure document-level access control documentation illustrates one Azure-specific, partly preview pattern in which permission metadata is projected to indexed chunks and applied during retrieval. Other architectures may enforce authorisation at different boundaries. The general requirement is a current authorisation decision for the subject and requested resource before retrieved content reaches model context; metadata alone is not enforcement.

For the permission-preserving retrieval detail, see preserve permissions in retrieval.

Minimise unnecessary sensitive information

Before a document enters the pipeline, ask whether it contains sensitive information that the task does not need. Removing unnecessary material reduces what the retrieval system can expose.

Minimisation follows the classification and governance decision. The classification owner decides what may enter the system and what should be removed or redacted. A parser should not make that decision silently.

For the classification and minimisation detail, see identify sensitive data before ingestion, which covers the classification framework and the minimisation decisions that precede preparation.

The NCSC's secure development guidance supports protecting, tracking and documenting relevant data, prompts and logs. This is high-level lifecycle guidance, not a RAG preparation standard or safety result.

Keep preparation separate from chunking experiments

Preparation and chunking answer different questions. Preparation decides what content enters the pipeline and how it is represented. Chunking decides how that prepared content is divided into retrieval units.

Keeping the hand-off clear has practical benefits:

• You can experiment with chunking strategies without re-preparing the source documents.
• You can inspect the prepared output independently of the chunking output.
• You can change the chunking strategy without affecting the preparation pipeline.

Use the prepared representation as the stable input when you follow the RAG chunking strategies guide. This lets you compare segmentation choices without quietly changing the source preparation at the same time.

Define acceptance tests, exceptions and evidence

Before ingestion, define and run acceptance tests that fit the source and use case:

• Parse fidelity. Does the parsed output preserve the source's structure and meaning?
• Metadata completeness. Are all required metadata fields present and correct?
• Permission projection. Are the source permissions correctly projected to the document's representation?
• Version accuracy. Is the exact version recorded and verifiable?
• Exception resolution. Are all exceptions resolved or documented with an owner?

Record the acceptance evidence, test version and any accepted exception. Give each exception an owner and a review trigger. Later, reviewers can inspect a decision rather than reconstructing it during debugging or change review. The record does not certify the document, parser or downstream system as safe.

Check the exact document version before ingestion

Once ownership, permissions, classification and preparation evidence are in place, check the exact file that will be sent to chunking and indexing.

.mdSiren is a document security workspace for AI. When it launches, Standard Scan will check that exact version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. This decision supports source readiness; it does not choose the source, grant access or validate the full RAG system.

See what Standard Scan checks.

For the document-specific prompt-injection signal detail, see check a document for prompt-injection signals.

Ingest, refresh, supersede and delete traceably

Once every applicable gate has passed, the document representation accepted for ingestion can enter the index. Those gates include source and use-case approval, provider assessment, classification and minimisation, current source permissions, preparation acceptance tests and any applicable exact-version checkpoint. The ingestion record should include:

• The document ID and exact version.
• The ingestion date and time.
• The parser and configuration used.
• The metadata attached.
• The acceptance evidence.

Keep refresh, supersede and delete operations traceable:

• Refresh. When the source document changes, the system detects the change, reprocesses the new version and updates the index. The old version is marked as superseded.
• Supersede. When a new version replaces an old one, the old version is marked as superseded and is no longer retrieved.
• Delete. When a document is withdrawn, the system removes or suppresses the content and its derived chunks. The deletion is recorded and verified.

The Azure RAG solution design guide supports evaluating ingestion and runtime stages plus the complete system using representative content and queries. Its topology is not universally optimal.

Frequently asked questions

How do you prepare documents for RAG? Select eligible sources, record ownership and provenance, parse and inspect the output, normalise content, add metadata, minimise unnecessary sensitive information, define acceptance tests and ingest traceably. Give each step an owner and retain its evidence.

Which document metadata should be retained? Provenance (source ID, version, owner, dates), permissions (access rules projected to chunks), lifecycle (ingestion date, refresh date, deletion flag, superseded-by) and classification (sensitivity label).

Should documents be cleaned before chunking? Normalise them where consistency helps, but do not silently remove meaning. Record every material transformation. This work belongs in preparation, before chunking experiments begin.

How do permissions reach a vector index? One pattern, illustrated by Azure's partly preview documentation, projects permission metadata from a source document to indexed chunks and applies it during retrieval. Other architectures may enforce authorisation at different boundaries. In every case, metadata alone is not enforcement: the system needs a current authorisation decision before retrieved content reaches model context.

When should a changed document be reprocessed? When the source document changes, the system should detect the change, reprocess the new version and update the index. A changed document is a new version to check.

Does chunking remove unsafe instructions? No. Chunking is not sanitisation. NIST's definition of indirect prompt injection refers to attacker-controlled external resources; the presence of instruction-like content alone does not establish attacker control or intent. Nor does it establish whether a model will follow that content. Keep three propositions separate: a signal was found in a document; a defined model or pipeline changed behaviour in a test; and a consequential outcome occurred. Evidence for one is not proof of the next.