This guide takes a cross-functional owner from a defined task to a controlled, evaluated knowledge workflow.
The blueprint turns that route into explicit decisions and evidence. It is an editorial synthesis, not an official framework. Adapt the owners, artefacts and failure tests to your organisation's sources, risks and operating model.
| Stage | Decision | Accountable owner | Required input or evidence | Output | Failure test |
|---|---|---|---|---|---|
| Task definition | Which user task and decision will the system support? | Product owner | User research, task examples, escalation needs | Bounded task statement and success criteria | The team cannot distinguish an in-scope request from one that requires deferral |
| Source approval | Which source systems are eligible for that task? | Information owner | Source inventory, use approval, provider decision | Approved source register with exclusions | A source enters the plan without an owner, purpose or recorded approval |
| Source permissions | Who may use each source and under which conditions? | Information owner and security lead | Current access policy, subject and resource attributes | Permission model and negative test cases | A user can retrieve content unavailable to them in the source system |
| Preparation | What must be parsed, normalised, minimised and inspected? | Data owner | Representative exact versions, parser output, classification decision | Prepared source package with provenance and version record | Important content, structure or restrictions are lost without detection |
| Ingestion | How will prepared content become traceable derived artefacts? | Engineering lead | Accepted source package, chunking and metadata rules | Versioned chunks, metadata and index records | A derived record cannot be traced back to its exact source version |
| Retrieval | Which eligible passages should a request be able to retrieve? | Retrieval owner | Representative queries, permission-filter design, relevance criteria | Candidate passages with scores and source references | Relevant content is repeatedly missed or restricted content is returned |
| Context assembly | What information should reach the model for this task? | Application owner | Instructions, retrieved passages, history, budget and permission decisions | Bounded context package | Context is stale, contradictory, over budget or outside the user's permissions |
| Generation | What may the model answer, format or defer? | Product owner | Context package, response contract, refusal and escalation rules | Answer or explicit deferral | The system acts outside its decision boundary or hides material uncertainty |
| Evaluation | Which quality, access and operational behaviours must be tested? | Evaluation owner | Representative tasks, expected evidence, negative permission cases | Test results and known limitations | A material failure mode has no test or named reviewer |
| Release | What evidence is sufficient for a bounded launch? | Accountable system owner | Evaluation results, risk decisions, runbooks and residual limitations | Recorded release decision and operating scope | Release proceeds despite an unmet gate without an explicit risk decision |
| Refresh | What change triggers reacquisition, reprocessing and re-evaluation? | Operations lead | Source change events, review cadence, dependency versions | Updated corpus and regression evidence | Superseded content remains retrievable after its replacement should be active |
| Deletion | How will a withdrawn source and its derivatives be removed? | Information owner and operations lead | Source identifier, lineage record, caches and retention obligations | Removal evidence covering indexed and downstream copies | A known derived artefact remains discoverable after the removal procedure completes |
What is an internal AI knowledge base?
A RAG-based knowledge system retrieves relevant passages from an indexed corpus and supplies them to a generative model as context for a defined task. The original architecture described in Lewis et al. (2020) combined a sequence-to-sequence model's parametric memory with retrieved passages from a non-parametric index, motivated partly by provenance and knowledge-updating limitations. That paper is one influential design, not a universal blueprint.
Current implementation guidance, such as the Azure RAG solution design guide, separates ingestion and runtime stages and recommends evaluating both stages and the system end to end. Your implementation will differ in topology, model choice and orchestration; the split is a useful way to assign decisions and tests, not a universal architecture.
An internal AI knowledge base is not a single product, a single database or a single model. It is a governed workflow that connects approved sources to defined tasks through preparation, retrieval, generation and evaluation.
Start with the user task, decision boundary and owner
Before choosing technology, define:
- The task. What specific question or decision will the system support? "Answer employee questions about onboarding policy" is a task. "Be a company brain" is not.
- The decision boundary. What may the system answer, and what must it defer to a human? Which actions, if any, may it trigger?
- The accountable owner. Who is responsible for the system's behaviour, its source corpus and its release decisions?
The NCSC's secure design guidance supports threat modelling, least privilege and constrained actions from the outset. Defining the task and boundary first makes those controls concrete rather than abstract.
Approve the use case, provider and information scope
Governance establishes the purpose, ownership and limits within which technical controls operate. Before any document enters the system:
- Approve the use case. A named person or committee confirms the task is appropriate, the risks are acceptable and the owner is identified.
- Select the provider and model. Record the model, its version, its data-handling terms and any relevant contractual constraints.
- Define the information scope. Which categories of information may enter the system? Which are excluded? This is a classification and minimisation decision, not a technical one.
The ETSI baseline specification TS 104 223 sets requirements for threat and risk management, access control and auditability across the AI system lifecycle. It is a baseline, not a certification, but it provides a useful checklist for the governance layer.
Select source systems and preserve provenance
Identify the source systems that will feed the knowledge base: document repositories, wikis, policy stores, ticketing systems, or other structured and unstructured sources. For each source, record:
- The system name and owner.
- The access method and credentials (stored securely, not in the index).
- The update frequency and change-detection mechanism.
- The provenance: who created or last modified each document, and when.
Provenance supports traceability, deletion work and decisions about whether a retrieved passage is current. It does not make the source trustworthy or the content correct. The Azure RAG design guide treats document preparation, metadata and persistence as ingestion concerns, and provenance belongs in that layer.
Choose the retrieval and context design
The retrieval design determines how a user query becomes a set of relevant passages. Key decisions include:
- Index type and embedding model. These affect retrieval quality but are workload-specific; there is no universal best choice.
- Retrieval count and reranking. How many passages are retrieved, and are they reranked before context assembly?
- Context assembly. How are retrieved passages combined with instructions, system prompts and any interaction history?
At the context layer, engineer the agent's context so that the information supplied at inference time is compact, high-signal and task-relevant. Retrieval feeds context assembly, while context engineering deals with selecting and maintaining that information.
The overall retrieval and ingestion flow is covered in detail in the RAG pipeline architecture guide, which maps the complete two-lane system from source approval through generation and evaluation.
Prepare, ingest and version company documents
Do not send documents straight to the index. First parse and normalise them, add metadata, record the version and check their eligibility. Treat preparation as a distinct stage with its own decisions and failure modes.
The document preparation guide covers the full pre-ingestion checklist: source selection, ownership and permission checks, parsing inspection, metadata, version records and acceptance tests. A successful parse is not proof that the source is authorised, current or complete.
Treat every document change as a new version. If the system indexes version 3 of a policy and version 4 is published, it must detect the change, reprocess the document and update the index. Otherwise, stale content becomes a correctness and trust problem.
Preserve permissions through retrieval and generation
A vector index does not automatically carry source permissions. If a document is restricted to a specific team, that restriction must survive chunking, indexing and retrieval. The Azure document-level access control documentation illustrates one implementation in which permission metadata is projected to indexed chunks and applied during retrieval. That implementation is Azure-specific and partly in preview; permissions can be absent, stale or misapplied in any system.
The NIST Zero Trust Architecture publication supports protecting resources without trusting network location alone. In a knowledge base, this means that authorisation should be tied to the requested resource and the current user, not to the fact that the request came from an internal network.
Permission preservation is a specialised implementation job. Specify the identity-to-retrieval authorisation path, how policy changes reach derived artefacts, and how revocation is tested rather than assuming the index will reproduce the source system's behaviour.
Evaluate quality, access behaviour and operational outcomes
Do not reduce evaluation to one score. The Azure RAG LLM evaluation guide treats groundedness, completeness, context utilisation, relevance and correctness as distinct, workload-dependent dimensions. An answer can be well grounded in supplied context while still being incorrect. No single metric or threshold proves safety or factual accuracy.
Before release, evaluate:
- Retrieval relevance. Does the system retrieve the right passages for representative queries?
- Answer faithfulness. Does the generated answer follow the supplied context?
- Factual correctness. Is the answer factually accurate, independent of whether it is grounded?
- Access behaviour. Does the system correctly exclude restricted content for users without permission?
- Operational outcomes. Does the system reduce the time or effort for the defined task?
The RAG evaluation guide covers test sets, metrics, release gates and regression monitoring in detail.
Check the exact source version before AI use
Once you have approved the use case, provider, source, information scope and permissions, check the exact document version before it enters the knowledge base. This creates a clear hand-off between governance decisions and document intake.
.mdSiren is a document security workspace for AI. When it launches, Standard Scan will check the exact version for supported prompt-injection and document-borne risks, route it to Approved, Needs review or Quarantine, and keep the Approved exact version in your private Library. If the file changes, check the new version. This supports document intake; ownership, classification, permissions, ingestion and system security remain part of the wider knowledge-base workflow.
For wider document-to-AI use cases and the document-checking workflow, see document-to-AI use cases.
Operate the system: refresh, deletion, monitoring and incidents
The work does not stop at launch. Operational responsibilities include:
- Refresh. Detect source changes, reprocess affected documents and update the index. Define the refresh cadence and the owner.
- Deletion. When a document is withdrawn or a user requests removal, the system must delete or suppress the affected content and its derived chunks. Verify that deletion is complete, including in caches and logs.
- Monitoring. Track retrieval quality, answer feedback, permission errors and unusual access patterns. Define which changes trigger regression testing and who may accept the resulting residual risk. The ETSI baseline specification spans maintenance and end-of-life controls as well as access control and auditability, but it does not show that a particular monitoring design is effective.
- Incidents. Define what constitutes a security or quality incident, who is notified, and how the system is taken out of service or restricted during an investigation.
The RAG security guide covers the system-wide threat model and layered controls that underpin these operational responsibilities.
A phased build roadmap
This phased roadmap is an editorial synthesis, not a universal calendar. Adapt its sequence and duration to your organisation.
| Phase | Owner | Output | Test of completion |
|---|---|---|---|
| 1. Task and governance | Product owner and security lead | Approved use case, decision boundary, information scope, provider selection | Signed-off use-case document; risk assessment complete |
| 2. Source selection and preparation | Data owner and engineering lead | Approved source list, preparation pipeline, versioning mechanism | Sample documents prepared and inspected; provenance recorded |
| 3. Ingestion and indexing | Engineering lead | Index populated with prepared documents, metadata and permission projections | Retrieval returns expected passages for test queries; permissions verified |
| 4. Context and generation | Engineering lead and product owner | Context assembly, model interface, output formatting | Representative tasks produce acceptable answers; faithfulness checked |
| 5. Evaluation and release gates | Evaluation owner | Test sets, metrics, release criteria, regression baseline | All release gates pass; access-control tests pass |
| 6. Operational handover | Operations lead | Monitoring, refresh, deletion and incident procedures documented and tested | Simulated refresh and deletion succeed; incident runbook exercised |
Each phase has a named owner, a concrete output and a testable completion criterion. Treat the tests as dependency gates where a later activity relies on the earlier output. Compatible work may run in parallel, and a team may revisit an earlier decision when evaluation exposes a weak assumption; record the exception rather than forcing a universal waterfall.
Frequently asked questions
What is an internal AI knowledge base? It is an organisation-specific system that makes approved internal knowledge available to defined AI-assisted tasks through retrieval, context assembly and generation. It is not a single product or a single model.
Is a company brain the same as RAG? "Company brain" is shorthand for an organisational knowledge system. RAG is one retrieval pattern that such a system may use. A company brain also includes governance, permissions, evaluation and maintenance, which go beyond the retrieval mechanism.
Which documents should enter an AI knowledge base? Only documents that have been approved for the specific use case, classified for sensitivity, minimised where appropriate, and whose permissions can be preserved through retrieval. The source owner or policy decides eligibility.
How do you keep an AI knowledge base current? Define a refresh mechanism that detects source changes, reprocesses affected documents and updates the index. A changed document is a new version to check. Monitor for stale content and set a review cadence.
How should permissions work? Where the retrieval layer relies on permission metadata, carry the relevant source attributes to derived chunks and apply a current authorisation decision when serving results. Network location alone should not confer trust. Test both permitted and denied cases, including revocation, before release.
How do you test it before release? Use representative test sets to evaluate retrieval relevance, answer faithfulness, factual correctness, access-control behaviour and operational outcomes. Set workload-specific release gates and rerun evaluation after changes.
