RAG: From Working Pipeline to Governed Deployment
The Backstory
I had been using Claude daily for months. Drafting, thinking, building, sparring. Getting genuine value from it. The outputs were good. The process felt natural. And somewhere in that fluency I had developed a working mental model of how these systems behaved.
What I had not done was build one. The actual integration layer. The pipeline that takes a body of knowledge, prepares it for retrieval, and connects it to a language model in a way that is reliable, governed, and honest about its own limitations.
That gap between using AI fluently and understanding how to deploy it at enterprise scale is the gap I am closing deliberately. One build at a time.
The RAG build was a specific exercise in that learning. I used The Architect’s Field Guide as the corpus. A 65,000 word practitioner guide for architects navigating complex organisational situations, in its final stages before launch at the time of writing. Fifty situations. Six zones. Consistent structure throughout. A clean corpus with clear semantic boundaries that would let me observe the RAG pattern without enterprise data complexity getting in the way.
The enterprise data complexity comes later. First you need to understand the pattern itself.
What RAG Is and Why It Matters
A large language model does not know your business, your organisation, or your processes exist.
It was trained on a vast body of general knowledge and it answers from that training. Ask it about your internal credit policy, your regulatory obligations in a specific jurisdiction, your architecture decisions from the last three years, and it will answer from the closest approximation it can construct. That approximation may be plausible. It may even sound authoritative. The output is not your organisation’s knowledge.
Retrieval Augmented Generation is the pattern that solves that problem. You prepare your documents, make them searchable by meaning rather than keyword, and when a question arrives you find the relevant passages and give them to the language model alongside the question. The model answers from what it has been given. The generation is grounded in your knowledge rather than borrowed from the internet.
That grounding is what makes AI-first process transformation possible at enterprise scale. An AI system operating inside a business process without it is reasoning from general knowledge about a specific organisational context it has never seen. With it the system reasons from the organisation’s own policies, decisions, process documentation, and institutional knowledge. An AI system embedded in how the organisation operates rather than performing a sophisticated approximation of it. Or, less charitably, an expensive search engine that occasionally hallucinates.
RAG is not a tooling choice. It is a foundational architecture decision. The consequences of getting it wrong are not immediately visible. They surface later, in production, when the conditions the demonstration never tested finally arrive.
What I Built and How
The build used three tools. The Anthropic SDK for the generation layer. Voyage AI’s voyage-3 embedding model accessed through the Anthropic SDK. ChromaDB as the local vector store. Python to connect them. Claude Code to write the scripts. The entire pipeline ran locally with the Anthropic API and a separate Voyage API key as the only external dependencies.
The corpus was 65,000 words structured around fifty situations across six zones. That structure was the first decision point.
Chunking
The field guide’s own structure was the guide. Each situation was a self-contained unit of meaning. Using the situation boundaries as the chunk boundaries preserved semantic coherence. The structure of the document told us where to cut.
The alternative, splitting on fixed token counts at regular intervals, would have cut across the natural boundaries of meaning. A situation split mid-argument. A lever extracted without the trap it was designed to address.
Where you cut determines what the system can retrieve. A chunk that has lost its context cannot be retrieved meaningfully even if the words are present. Getting chunking wrong at indexing time means rebuilding the entire index to fix it. It is an architectural decision with that kind of consequence.
Embedding
Each chunk was converted into a vector using Voyage AI’s voyage-3 embedding model, accessed through the Anthropic SDK. A vector is a list of 1024 numbers that together represent the meaning of the text. Passages with similar meaning produce similar vectors. That is what makes semantic search possible. You are searching by meaning, not by keyword.
The same model must be used for both indexing and querying. Different models produce vectors in different spaces. You cannot mix them. If the embedding model is ever changed, the entire index has to be rebuilt. That is a governance constraint with real operational implications.
The Vector Store
The vectors were stored in a local ChromaDB collection along with the chunk text and metadata. When a question arrives, the system converts the question into a vector using the same model, then finds the stored chunks whose vectors are most similar. The similarity is measured as distance. Closer means more similar meaning.
Every chunk came back with a similarity score. That score is a signal about the reliability of what follows. Legitimate queries scored between 0.79 and 0.99. An out-of-scope query scored 1.57 to 1.61. Nearly double. The score can become a reliable signal regarding where the response would land.
Retrieval and Generation
The three closest matching chunks were passed to Claude alongside the original question. The system prompt instructed Claude to answer only from the provided context and to say so clearly if the answer was not there. Claude read the chunks and answered from what it was given.
That boundary was tested directly. A question about a car mechanic scam was put to the system. The similarity scores immediately signalled that nothing relevant existed in the corpus. The system prompt boundary held. The model declined to answer and explained why. That behaviour was designed through a specific system prompt instruction. In a deployment where nobody thought to design for out-of-scope queries, it would not have held.
The Metadata Decision
The first version of the index stored only chunk text and a title. The metadata was thin. Zone, situation number, character count, a structured chunk ID, none of it was there. Nothing that would allow filtering or governance of what the system retrieved.
The index was rebuilt with all of those fields. They become the dimensions along which retrieval can be scoped and governed.
In an enterprise deployment those fields reflect governance requirements. Jurisdiction, effective date, version, document owner, approval status. The metadata schema is a decision made before the first document is indexed. Like chunking strategy, it is painful and disruptive to change later.
What the Build Actually Taught Me
Chunking has to be process context-aware.
The build used the field guide’s situation boundaries as chunk boundaries. Retrieval quality was immediately better for it. The insight goes deeper than document structure though. It is about the business process the system serves.
A chunk is the unit of meaning the retrieval system will present to a language model as the basis for a decision or an action inside a business process. The boundary of that chunk needs to reflect the boundary of meaning the process depends on. A policy clause that only makes sense with the definition it references. A process step that is only valid under the condition that precedes it. A regulatory obligation that is only applicable in the jurisdiction stated two paragraphs earlier.
Structure-aware chunking uses the document’s formatting as the guide. Process context-aware chunking asks what unit of meaning the actor in this process actually needs. Related questions. Different answers.
The similarity score is a control signal.
In the build, legitimate queries scored between 0.79 and 0.99. The out-of-scope query scored 1.57 to 1.61. The score signalled the problem before the generation step began. That signal was used to decline the answer through the system prompt boundary.
In a production deployment the signal has three distinct uses. It can flag a response for human review when the score falls in an uncertainty band. It can block an automated process from proceeding when the score exceeds the threshold for reliable retrieval. It creates an auditable record of every query, every retrieval score, and every decision made on the basis of that score.
A similarity score used only to select chunks is a retrieval mechanism. Used to flag, block, and audit, it is a control.
The confidence threshold is corpus-specific and by extension process-specific.
The threshold ranges that define the control bands have to be derived from the specific corpus they govern, using a labelled evaluation set that reflects the real range of inputs the process will encounter. The score distribution of a regulatory policy corpus will be different from a process documentation corpus. The thresholds cannot be transferred between deployments.
The corpus represents a specific process context. So the threshold is process-specific by extension. The process evolves, the documents change, the score distribution shifts, the threshold needs revalidation. In a regulated environment that revalidation is an audit obligation with a named owner and a review cycle.
Chunk metadata is to RAG what event metadata is to event-driven architecture.
In event-driven architecture, event metadata carries the business context that downstream consumers depend on. The event without its metadata is a technical signal with no business meaning attached. The consumer cannot route it, govern it, or audit it without knowing what it represents in business terms.
Chunk metadata is the same thing for retrieval. Text in an index without metadata is ungovernable. With metadata it becomes a governable unit: scoped to a jurisdiction, tied to a policy version, owned by a function, valid within a date range. The metadata schema is the governance architecture of the index. Like event schema design, it is a decision made before the first message is published. Changing it later is disruptive by design.
The system prompt is the system contract.
In API design, a contract defines what the system is permitted to do, what it must decline, and what it returns under what conditions. The system prompt in a RAG deployment is that contract between the deployment and the model. It defines permitted behaviour for every query, what the model must decline, and what it returns when the answer is not in the retrieved context.
In the build, that instruction is what made the out-of-scope boundary hold. Without it the model would have answered from general knowledge and presented it as if it came from the corpus.
In a regulated deployment that contract needs to be precise, tested, version controlled, and owned by the appropriate function. A system prompt left as a developer implementation detail is a governance gap.
When RAG Meets the Enterprise
The build used a clean corpus and a single well-defined question space. The field guide covers fifty situations that architects face. The retrieval system was designed around that scope. The corpus was coherent, consistently structured, and entirely within a single domain.
An enterprise knowledge base is none of those things.
The more fundamental challenge is a question the build did not have to answer because the corpus was clean and the question space was bounded. In an enterprise deployment it becomes unavoidable.
What is this system actually serving?
A RAG deployment is a system that grounds a language model in a specific body of knowledge so that a specific actor can do a specific thing inside a specific process more reliably than they could without it. Every design decision, the corpus, the chunking strategy, the metadata schema, the confidence threshold, the system contract, flows from that specificity.
The business process is the unit of functionality. The specific process, with its actors, its steps, its decision points, and its outputs.
The design starts with the process. What does this actor need to do their job reliably. What questions do they need answered. What decisions do they need to make. What level of confidence do they need before they can act on an output without independent verification.
The corpus is the bounded context for that process. The documents that represent the knowledge the process depends on. Scoped explicitly. Owned by the function that owns the process. Reviewed on the same cycle as the process itself.
The chunking strategy reflects the semantic units the process depends on. The units of meaning the actor in the process actually needs to retrieve intact.
The confidence threshold is derived from the questions that actor actually asks in the context of that process, against the corpus that represents it. A process-specific control.
The system contract defines what the system is permitted to answer within the scope of that process, what it must decline, and what happens at the boundary.
The audit trail is the record of every time that contract was exercised. Every query. Every retrieval. Every answer. Every decline. Auditable at the level of granularity the regulatory environment requires.
A RAG deployment designed this way is a process component. The infrastructure that makes it possible for an AI-first process to operate reliably, within defined boundaries, with the governance controls in place before production rather than retrofitted after the first failure.
That is what enterprise scale actually requires. A deployment designed around the process it serves, with every component derived from what that process needs.
Where This Leaves Me
I started this build with one question. Does understanding RAG conceptually translate into knowing how to deploy it reliably?
The answer is no.
The tools are accessible, the documentation is good, and a clean corpus produces results that look entirely capable in a controlled environment. That part took one session.
What the build revealed is everything that has to be explicitly designed before a working pipeline becomes a deployment you can trust inside a real process. The chunking strategy shaped by process context. The metadata schema that makes retrieval governable. The confidence threshold derived from the corpus. The system prompt owned as a governance contract. Every one of those things had to be a deliberate decision.
A RAG deployment designed without a named process and a named actor is a demonstration. Designed around a specific process, with every component derived from what that process needs, it is a foundation.
The next article takes that as its starting point. What it actually means to design a RAG deployment around a business process. What changes when the business process is the unit of functionality. What that asks of the people responsible for making it work.
The pipeline is the easy part.
