What does local RAG mean?
"Local RAG" is not a standard term with one fixed technical definition. It is a working label for a RAG system whose named components are deployed on infrastructure that the operator controls for a stated purpose. Its meaning depends on the boundary you draw and what "local" means in that setting.
The NIST definition of cloud computing says a private cloud may be operated by the organisation, a third party or both, and may exist on or off premises. So "private" does not automatically mean "local" or "on premises".
The Hugging Face Transformers installation documentation gives one concrete example. Offline model use requires the needed files to be obtained or cached in advance and the library to be placed in offline mode. Running code locally and stopping runtime network retrieval are separate operational steps. This is a product-specific example, not a universal definition.
Local, self-hosted, on-premises, private and offline compared
| Property | Typical operator | Physical location | Network condition | What it does not establish | Unresolved check |
|---|---|---|---|---|---|
| Local execution | The organisation or an individual user | A named local endpoint or machine | May still be connected | Offline, private, single-tenant or free of telemetry | Are there external API calls, telemetry or update checks during the defined task? |
| Self-hosted | The organisation or its infrastructure provider | On premises or remote | Connected, restricted or offline | On-premises operation or security | Which party operates each component, and which external dependencies remain? |
| On-premises | The organisation, sometimes with supplier support | The organisation's premises | Connected, restricted or isolated | That every data path remains on premises | Do inference, logs, monitoring or support tools call an external service? |
| Private cloud | The organisation, a third party or both | On or off premises | Normally connected within defined boundaries | Local, offline, air-gapped or inherently secure | Who operates it, where is it hosted and which network boundaries are enforced? |
| Offline operation | The operator of the named workload | Anywhere the complete workload runs | No network calls during the stated operation | Air-gapping or freedom from previously downloaded dependencies | Were models, packages and data cached first, and was runtime traffic observed? |
| Air-gapped | The organisation controlling the isolated environment | A physically controlled environment | No ordinary external network path; controlled transfer routes may remain | Protection from internal threats, removable media or unsafe processes | Which transfer routes remain, who can use them and how are imported artefacts checked? |
These are working boundary definitions, not a claim that the industry uses one settled taxonomy. Use the labels only after recording the operator, location, network condition and remaining dependencies for your actual build.
Define the workload, users and constraints
Before selecting components, write down:
- Workload. What tasks will the system perform? What is the expected query volume and latency requirement?
- Users. Who will use the system? What are their permission boundaries?
- Constraints. What are the deployment, network, storage, compute and compliance constraints?
- Data sensitivity. What is the most sensitive data the system will process?
This definition tells you which deployment properties matter and which checks the build must pass.
Turn those requirements into testable statements before choosing a stack. For example, replace “the system must be private” with “during an agreed test window, no document content leaves these named hosts through these named interfaces”. Record who owns each statement, how it will be observed and what result would block deployment. A one-line boundary statement is useful here: name the workload, users, components, hosts and allowed connections. This stops a procurement label becoming an untested security claim.
Map a vendor-neutral local RAG system
The following reference build is an illustrative architecture, not a universal implementation. It shows the jobs that need owners and boundaries without prescribing a stack.
| Component | Responsibility | Deployment consideration |
|---|---|---|
| Source acquisition | Obtain documents from approved sources | Network access to source systems; authentication |
| Preparation | Parse, normalise, add metadata | CPU; storage for intermediate output |
| Chunking | Divide content into retrieval units | CPU; configuration per corpus |
| Embedding | Convert chunks to vector representations | CPU or GPU; model storage; offline model files |
| Index | Store vectors and metadata for retrieval | Storage; memory; persistence |
| Retrieval | Query the index for relevant passages | Latency; permission filtering |
| Context assembly | Combine passages with instructions | Token budget; ordering |
| Model inference | Generate answers from context | CPU or GPU; model size; offline model files |
| Interface | Accept user queries and return answers | Network exposure; authentication |
| Logs | Record queries, retrievals, answers, errors | Storage; retention; access control |
| Evaluation | Test retrieval and answer quality | Test set; evaluation criteria |
You can start with the generic RAG pipeline to place this local build inside the wider ingestion and runtime flow.
Prepare and ingest a small, approved corpus
Start with a deliberately small but representative corpus of approved documents. Test the full flow before scaling it. Size alone is not the selection rule: the set needs to exercise the formats, structures, permission patterns and update behaviour expected in use.
- Select enough documents to represent the expected content types, including awkward examples such as long sections, tables and a document with restricted access.
- Record provenance, version, owner and permissions for each.
- Parse and inspect the output.
- Chunk using a starting strategy appropriate to the document structure.
- Embed and index.
- Verify that a sample of retrieved passages traces to the correct source and version.
Keep the original files and derived records connected by stable identifiers. For each document, record the parser and configuration, resulting chunks, embedding model and index build. If one input changes, the old result becomes comparison evidence. It does not show that the new build behaves in the same way.
You can prepare documents for ingestion using a detailed pre-ingestion checklist.
Retrieve context and generate an answer
Now test the runtime flow:
- Submit a representative query.
- Verify that retrieval returns relevant passages.
- Verify that permission filtering excludes unauthorised content.
- Verify that the generated answer follows the supplied context.
- Verify that attribution links the answer to its sources.
- Record the full trace: query, retrieved passages, context, answer, latency.
For one test case, follow the trace in both directions. Start with the answer and identify the assembled passage, retrieved chunk and exact source version. Then start with that source version and confirm which chunk and index record were created. A missing link exposes a provenance problem even when the answer reads well. Repeat the trace with an unauthorised user and with a query that should return no supported answer.
Preserve permissions and inspect secrets, network calls and logs
Source permissions still need to be enforced at retrieval time. The NIST Zero Trust Architecture publication supports protecting resources without implicit trust based on network location. For a local RAG design, the practical lesson is simple: running the system on your infrastructure does not authorise a user to retrieve every document.
You can preserve access controls using a structured permission-preserving design.
Inspect the system's network calls, secrets and logs rather than inferring their behaviour from the deployment label:
- Network calls. What external calls does the system make? Are there telemetry, update or API calls?
- Secrets. Where are API keys, model credentials and database passwords stored? Are they accessible to the process?
- Logs. What is logged? Where are logs stored? Who can access them?
The NCSC secure deployment guidance supports access control, incident planning, testing and clear communication of system limitations.
Verify the complete data flow
The checklist below is an editorial synthesis. Complete it for a defined build and observation period. Here, “verified” means the operator retained evidence of the stated test. It is not a general security certification.
| Path or component | What to observe or test | Evidence to retain | Question that must remain open if untested |
|---|---|---|---|
| Source storage and preparation | Where originals, parsed text and temporary files are read and written | Configuration snapshot, file locations and an observed test trace | Can temporary or failed jobs write elsewhere? |
| Embedding and model runtime | Every endpoint contacted while ingesting and answering | Network capture or equivalent endpoint log tied to the test window | Does a fallback route call a remote service? |
| Model and dependency retrieval | How weights, packages, containers and licences arrive | Versions, hashes where available, origin and import record | What happens after a cache miss or restart? |
| Telemetry and support tooling | Metrics, crash reports, traces and administration connections | Enabled settings plus observed destinations | Can content, prompts or identifiers enter telemetry? |
| Updates | Automatic and manual routes for code, models and source content | Update procedure, approval record and rollback test | Which component can update without review? |
| Logs | Query, retrieved context, answer and error fields, including redaction | Log schema, sample record and access test | Do exceptions capture more content than normal events? |
| Backups and replicas | Copies of sources, indexes, logs and configuration | Inventory, destination, access test and restore result | Are snapshots or replicas outside the stated boundary? |
| Revocation and deletion | Removal from sources, chunks, indexes, caches, logs and backups | Test case with timestamps and any documented residual copy | When does each derived or backed-up copy cease to be usable? |
Run the checklist during ordinary operation and defined failure conditions. One clean query does not cover model download, recovery, support access or an update job that runs on a different schedule.
Evaluate retrieval and answer behaviour
Before moving beyond a controlled test, evaluate the system against representative tasks. The Azure RAG design guide recommends evaluating ingestion and runtime stages as well as the complete system.
You can define RAG evaluation gates using a structured test-set and release-gate method.
Measure retrieval relevance, answer faithfulness, factual correctness, permission behaviour, latency and cost separately. No single metric proves quality or safety.
Use separate release gates for ingestion, retrieval, answer behaviour and access control. A build should not pass simply because an aggregate score hides one failed permission case. Retain the corpus version, queries, expected evidence, configuration and observed failures so that the same cases can be repeated after a model, dependency, parser or index change.
Plan updates, backups, revocation and deletion
A local RAG system needs maintenance after the demo works. Plan for:
- Updates. How are model weights, dependencies and source documents updated? What is the update procedure and rollback path?
- Backups. What is backed up, how often, and where? Is the backup accessible and testable?
- Revocation. How is a user's access to specific documents revoked? Is the revocation tested?
- Deletion. How is a document removed from the index? Is the deletion complete and traceable?
The NCSC secure operation and maintenance guidance supports monitoring and evaluating versioned changes because updates can alter behaviour.
Assign an owner and test interval to each process. Exercise restoration instead of inferring it from the existence of a backup. Test revocation across retrieval results and caches, not only at the source repository. Where retention rules prevent immediate removal from a backup, record that residual state and the controls that limit its use.
Why local is not automatically private or secure
Deployment location is a property, not a guarantee. A local system can still:
- Make external network calls for model updates, telemetry or API access.
- Store logs or data in a location accessible to unauthorised users.
- Use a model that was trained on data you do not control.
- Have a vulnerability in a dependency that exposes data.
- Be accessed by an unauthorised user through a misconfigured interface.
The NIST private cloud definition makes clear that a private cloud may be operated by a third party and may exist off premises. The Hugging Face offline-mode documentation gives one implementation example in which local files and offline operation are separate conditions.
You can threat-model RAG security to understand the full risk surface of a RAG system regardless of deployment location.
Where a source-document checkpoint fits
Deployment location answers where components run. A document checkpoint answers whether a particular source version should enter them. Treat these as separate decisions.
.mdSiren is a document security workspace for AI. When it launches, Standard Scan will check the exact document version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. This adds control at document intake; the RAG system still needs its own privacy, access, deployment and security controls.
Explore the document-checking workflow.
Frequently asked questions
What is local RAG? It is a working description for a RAG system whose named components run on infrastructure controlled for a stated deployment. It is not a standard term with one fixed technical definition.
Is local RAG the same as self-hosted RAG? Not necessarily. Self-hosted means the operator runs the service. Local means inference runs on a named local endpoint. A self-hosted service can run on remote infrastructure.
Can a local RAG system still send data over the internet? Yes. A local process can make network calls for model updates, telemetry, API access or dependency retrieval. Verify the complete data flow before calling the system offline.
Does running RAG on premises make it secure? No. On-premises is a physical location property. Security depends on access control, threat modelling, evaluation, monitoring and incident planning, regardless of location.
What should be tested before a local RAG system is used with company documents? Test the complete data flow: network calls, model dependencies, storage, logs, backups, permission enforcement and the full ingestion and runtime pipeline. Retain evidence for each path rather than relying on the word “local”.
