Define the RAG system boundary
The system boundary determines what is in scope for the threat model. Start by drawing the path from an approved source to the person or process receiving an answer. Include:
- Source systems and their permission models.
- The ingestion pipeline: acquisition, parsing, chunking, metadata, embeddings, indexing.
- The runtime: identity, query handling, retrieval, permission filtering, context assembly, generation.
- Tools and downstream actions, where the system can invoke them.
- Outputs: generated answers, cached results, logs.
- Operational infrastructure: storage, network, monitoring, update paths.
The Azure RAG design guide supports separating ingestion and runtime stages. Its topology is not a universal design or security control.
Write down what sits outside the box too. Identity providers, source applications, hosted models, observability services and human approval queues may all affect the result while being operated elsewhere. Record the data and control signals that cross each boundary, who operates the receiving component and what should happen if it is unavailable or returns an unexpected response.
The following lifecycle threat model is an editorial synthesis. Use it as an illustrative control map, then replace its generic questions with your own components, identities and failure conditions.
| Lifecycle stage | Asset or boundary to examine | Harmless scenario to test | Control and evidence question |
|---|---|---|---|
| Source systems | Document, owner, version and source permissions | A superseded policy is still marked current | Who approves the source, and can you show the approved version and permission record? |
| Acquisition | Connector, import job and source-to-pipeline boundary | An import silently misses several files | Do completeness checks, connector logs and retry records reveal the gap? |
| Parsing | Extracted text, structure and active document surfaces | A table heading is lost during conversion | Can a reviewer compare the parsed representation with the exact source version? |
| Chunking and metadata | Chunks, provenance, classification and permission fields | A chunk loses the section heading that gives it meaning | Can every chunk be traced to its source and checked for required metadata? |
| Embeddings and index | Vectors, search fields, replicas and index version | An old chunk remains in a serving replica | Can you identify which source and index versions are active, then remove or roll them back? |
| Query and retrieval | Caller identity, query, filters and ranked results | A valid user receives a result outside their current rights | Is current authorisation resolved and enforced before results are returned? |
| Context assembly | Selected passages, instructions, limits and ordering | Relevant evidence is omitted while an irrelevant passage is included | Can you reconstruct what was eligible, selected and excluded for this request? |
| Model inference | Model version, supplied context and generated response | The answer follows an instruction-like passage instead of the task | Can you reproduce the defined test and distinguish a document signal from a behaviour change? |
| Tools and actions | Tool contract, user authority and approval boundary | A read-only task requests a write action | Does deterministic authorisation block the action outside the model, and is the decision logged? |
| Outputs and caches | Answer, citations, cache key and audience | A cached answer outlives a permission change | Can you invalidate affected outputs and show which users or processes received them? |
| Logs and monitoring | Events, alerts, access to logs and retention policy | A retrieval filter fails without an alert | Are the necessary events present, protected and tied to a named response owner? |
| Deletion and re-indexing | Source, chunks, replicas, caches and derived artefacts | A withdrawn document remains retrievable | Can you trace removal through every copy and prove the serving system changed? |
Identify assets, actors and trust boundaries
Assets: source documents, derived chunks, metadata, embeddings, indexes, model and prompt artefacts, user data, generated answers, logs, credentials and approval records.
Actors: authorised and unauthorised users, source owners, system processes, administrators, reviewers, third-party services, models and tools. A model is not a security principal, but modelling it as an actor helps expose where probabilistic output can influence a deterministic decision.
Trust boundaries: source to ingestion, ingestion to index, identity to retrieval, retrieval to context, context to model, model to output, model to tool, output to user and system to external service.
The NIST Zero Trust Architecture publication supports protecting resources without implicit trust based on network location or asset ownership. It is not RAG-specific and does not prescribe a retrieval filter. For each boundary, note the identity presented, the resource requested, the decision point, the metadata relied on and the evidence produced. That turns a box-and-arrow drawing into a set of testable assumptions.
Protect source integrity and consider knowledge poisoning
Source integrity starts with a basic question: are the documents in the index the versions, content and permission sets the team intended to approve? Before discussing controls, prepare and govern source documents through a defined source register, owner, approval rule and change path.
Useful controls include allowlisting source systems, recording provenance, reviewing material changes, keeping an exact-version identifier and refusing an import when required ownership or permission metadata is missing. None of these proves that a document is true. They make the intake decision visible and give the team something concrete to inspect when an answer is challenged.
Knowledge poisoning research considers cases where content is introduced to influence later retrieval or answers. Keep that distinct from an ordinary quality error and from an instruction-like signal found in a file. The content alone does not establish who placed it, why it is there, whether it will be retrieved, whether model behaviour will change or whether any consequence will follow.
The PoisonedRAG paper treats the external knowledge database as an attack surface and reports attacker-chosen answer effects under specific poisoning experiments. The experiments are not evidence of attack prevalence, universal exploitability, real-world harm or failure of every defence. Do not detach a reported percentage from its corpus size, insertion budget, target questions, models and evaluation method.
Treat indirect prompt injection as a system risk
Indirect prompt injection occurs when an attacker controls a resource that a model processes, rather than the user input directly. The NIST glossary definition is based on attacker control of a resource. Instruction-like text in a resource does not by itself prove attacker control, malicious intent, model response, successful exploitation or consequential harm.
The Greshake et al. paper demonstrated, in defined synthetic and real-system experiments, that adversarial instructions retrieved from external content could alter model or application behaviour. These demonstrations do not establish population prevalence, applicability to every current model or architecture, a real-world incident, or a consequential outcome from merely finding similar text.
The BIPIA benchmark provides tasks in which instructions embedded in external content are tested for their ability to cause defined deviations in model outputs, and evaluates proposed defences within that benchmark. Benchmark success rates are not prevalence estimates or universal defences.
Our guide to understanding indirect prompt injection in documents covers the document-specific definition and checking guidance. Here, the important point is what the wider RAG system permits after that content is retrieved.
The NCSC blog on prompt injection explains that current language models do not provide a robust, deterministic separation between data and instructions. Its practical implication is residual-risk engineering: constrain privileged actions, add deterministic controls and monitor behaviour instead of claiming prompt injection has been eliminated. This is expert guidance, not proof that a particular passage is malicious or that one control will be effective in your system.
Prevent permission and retrieval leakage
Permission leakage occurs when content reaches a user or process that is not authorised to receive it. The central invariant is simple to state and easy to lose in implementation: current source rights must be resolvable and enforced before context assembly. Apply that rule to chunks, metadata, keyword fields, vector results, reranked results, citations and caches, not just the source document.
The NIST ABAC guide supports evaluating subject, object, action and environment attributes in an authorisation decision. It is not a ready-made index schema and does not prove that attributes are current or correctly enforced.
The Azure document-level access control documentation supports one permission-projection and retrieval implementation. Several native permission mechanisms currently use the 2026-08-01-preview API, and the documentation does not prove that source permissions reach retrieval correctly.
Our guide to designing permission-aware retrieval covers projection, filtering, change handling and tests in detail. In the threat model, retain the decisions and identifiers needed to connect a source permission change to every affected index record and cached output.
Constrain generation, tools and downstream actions
A RAG system that can invoke tools or take actions creates another control boundary. Retrieved content and generated text may help propose an action, but they should not grant authority to perform it. The NCSC secure design guidance supports least privilege, input checks and constraints on AI-triggered actions. It is high-level lifecycle guidance rather than proof that a particular design is safe.
Controls include:
- Expose only the tools required for the current task and user.
- Resolve user and service authority in deterministic application code.
- Validate tool arguments against a narrow contract before execution.
- Require a separate approval for consequential or hard-to-reverse actions.
- Record the request, authority decision, arguments, result and approver.
- Test denied paths as well as successful ones after any model, prompt or tool change.
Human approval helps only when the reviewer can see what will happen and has enough time and context to make a real decision. A vague confirmation box can become a rubber stamp. Show the target, scope, relevant source evidence and expected consequence, then fail closed if the approval cannot be obtained.
Address supply-chain and operational risks
Supply-chain questions include:
- Model weights from an untrusted source.
- Dependencies with known vulnerabilities.
- Update mechanisms that could introduce malicious code.
- Third-party services that process your data.
Operational questions include:
- Misconfigured access controls.
- Inadequate logging and monitoring.
- Missing incident response procedures.
- Stale or unpatched components.
The NIST adversarial machine-learning taxonomy provides a lifecycle taxonomy for adversarial machine-learning attacks and mitigations, organised by attacker goals, capabilities and knowledge, and records open mitigation challenges. NIST also records a potential erratum for the publication. A taxonomy is not a prevalence estimate, implementation standard or guarantee that a listed mitigation controls a particular system.
The ETSI baseline specification sets baseline requirements across secure design, development, deployment, maintenance and end of life, including threat and risk management, access control, provenance and auditability within its scope. It is not a complete secure-RAG design, legal certification or evidence that controls are correctly implemented.
For each dependency, record the version, provenance, operator, update channel, network path and rollback method. Then test the real deployment rather than relying on the label attached to it. A self-hosted component can still fetch remote code; a managed service can sit inside a tightly controlled data flow. The relevant question is what crosses the boundary and who can change it.
Apply layered controls across the lifecycle
The following risk/control/evidence/limitation matrix is an editorial synthesis. It helps assign work and expose residual risk. It is not certification, and the presence of a control is not evidence that the control works. Adapt the rows to the illustrative map above and retain results from defined tests.
| Asset or trust boundary | Harmless threat scenario | Layered controls | Evidence to retain | Owner | Residual limitation |
|---|---|---|---|---|---|
| Source system | Unauthorised document enters the index | Source approval, provenance recording, version verification | Approval log, provenance record | Data steward | Approval is a human decision; errors are possible |
| Ingestion pipeline | Malformed or corrupted document is ingested | Parse inspection, acceptance tests, conversion-loss review | Parse log, acceptance evidence | Data engineer | Conversion may lose information; not all loss is detectable |
| Index | Stale or unauthorised chunk remains retrievable | Permission projection, version tracking, deletion procedure | Index audit, deletion log | Platform engineer | Permissions can be stale; deletion may be incomplete |
| Retrieval | Unauthorised content is retrieved | Permission filtering, security trimming, tenant isolation | Retrieval log, filter decision log | Security engineer | Filter misconfiguration is possible; test regularly |
| Context assembly | Context includes unauthorised or irrelevant content | Context budget, permission-aware assembly, provenance tracking | Context snapshot | AI engineer | Context selection is heuristic; errors are possible |
| Model | Model generates a harmful or incorrect answer | Guardrails, constrained actions, human review for high-stakes tasks | Generation log, guardrail trigger log | AI engineer | Model behaviour is probabilistic; no guardrail is perfect |
| Tools and actions | Model triggers an unauthorised action | Tool permission scoping, human approval, action logging | Tool call log, approval record | Platform engineer | Tool scope may be too broad; test regularly |
| Logs and monitoring | Security event is not detected | Structured logging, alerting, regular review | Log retention, alert history | Operations lead | Logging may be incomplete; alerts may be missed |
| Updates and re-indexing | Malicious update introduces harmful content | Update approval, change testing, rollback procedure | Update log, test results | Platform engineer | Update testing may not cover all failure modes |
Use an AI governance framework to place these controls, assign ownership and set evidence and exception requirements. A useful review asks whether preventive, limiting, detective and recovery controls cover different failure modes, rather than counting how many controls appear in a register.
Place an exact-version source-document checkpoint
A threat model explains how the full RAG system will be protected. An exact-version document checkpoint adds one control at source intake.
.mdSiren is a document security workspace for AI. When it launches, Standard Scan will check the exact source version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. The result is one layer in a wider threat model; it does not predict model behaviour or secure the RAG system. See what Standard Scan checks.
Evaluate and monitor security behaviour
Security evaluation is distinct from answer-quality evaluation. The Azure RAG evaluation guidance supports distinct, repeatable evaluation dimensions. A quality metric is not a security verdict, and a well-grounded answer can still rely on incorrect or unauthorised context.
Security tests include:
- Permission enforcement: can a user retrieve content outside their current rights through direct search, generated answers, citations or caches?
- Adversarial behaviour: what happens in a defined harmless test when instruction-like content is retrieved, and which observed change is being measured?
- Tool scope: does deterministic authorisation deny a requested action outside the user's authority?
- Source change: does a changed, withdrawn or reclassified source propagate through indexes, replicas and caches?
- Log completeness: can the reviewer reconstruct identity, source version, retrieval decision, context, model version and any action?
You can evaluate security behaviour and regressions with representative test cases and declared pass conditions. Keep the input, expected decision, observed trace, result, reviewer and follow-up owner. A passing test supports only the defined system version and condition; it is not a general security verdict.
The NCSC secure operation and maintenance guidance supports monitoring and evaluating versioned changes because updates can alter behaviour.
Prepare incident, removal and re-indexing paths
An incident in a RAG system might begin with:
- Discovery that a document in the index contains harmful or unauthorised content.
- Detection that a user retrieved content they were not authorised to see.
- Identification that a model is generating answers that exceed its intended scope.
Prepare the path before you need it:
- Detect. How was the issue identified?
- Contain. Restrict retrieval or actions without destroying the evidence needed to investigate.
- Assess. Identify the exact source and system versions, affected identities, retrievals, outputs and actions.
- Remediate. Correct the source, permission, pipeline, policy or tool boundary that failed.
- Remove and re-index. Propagate the decision through chunks, indexes, replicas, caches and derived artefacts, then verify the serving path.
- Test. Repeat the relevant permission, adversarial, deletion and regression cases.
- Review. Update the threat model, owners, controls and exception decisions.
- Record. Retain a bounded account of what was observed and what remains unknown.
Avoid writing “the document caused the incident” when the evidence only shows that a signal was present. Record each step separately: what appeared in the source, what the defined system did after processing it, and what consequence was independently confirmed. That precision keeps the response useful without overstating attribution.
Frequently asked questions
What are the main security risks in RAG? The main families are source-integrity failures, knowledge poisoning, indirect prompt injection, permission leakage, over-broad tool authority, supply-chain changes and operational misconfiguration. Your actual priorities depend on the system boundary, users, actions and consequences.
Can a RAG knowledge base be poisoned? Research has demonstrated bounded poisoning experiments under specific conditions. This does not establish prevalence or universal exploitability. Controls include source approval, provenance tracking, version verification and monitoring.
How does indirect prompt injection affect RAG? If a document in the index contains instruction-like content, a model that retrieves it may change its behaviour in a defined test. This is a signal to review, not proof of an attack or harm. Controls include constrained actions, deterministic checks and monitoring.
Is access control enough to secure RAG? No. Access control is one layer. A complete security model also includes source integrity, input checks, constrained actions, monitoring, incident response and evaluation.
How should RAG security be tested? Use representative test cases covering permission enforcement, adversarial content, tool scope and log completeness. Repeat after changes to the system, corpus or model.
