Retrieval Augmented Generation Security: Keeping RAG Inside Your Compliance Boundary

Retrieval augmented generation is the architecture most regulated organizations reach for first, and for good reason. RAG lets you get value from a language model without training it on your data: documents stay in your repositories, get indexed, and are fed to the model as context only when relevant to a question. No fine-tuning, no shipping your corpus to a model vendor. That framing makes RAG sound inherently safer than training. It can be. It can also be the fastest way ever invented to hand your entire document estate to the wrong audience. Retrieval augmented generation security comes down to recognizing that a RAG system is not one thing. It is four or five data bearing components, and each one has to sit inside your compliance boundary and enforce your access model. 

The Index Is a Copy of Your Data, Treat It Like One 

The heart of a RAG system is the index: chunked text and vector embeddings derived from your source documents, stored in a search service or vector database. Two things about it matter enormously in a regulated environment. 

The chunks are your data, verbatim. Whatever sensitivity applied to the source document applies to the chunk store. If the source was CUI, the index processes and stores CUI, which makes it a CUI Asset under CMMC scoping and pulls it fully into your compliance boundary. 

The embeddings are your data too, just transformed. Embeddings are mathematical representations of content, and research has repeatedly shown that meaningful information can be reconstructed from them. Treating vectors as anonymized or de-identified is wishful thinking DoD or a regulator is unlikely to share. Encrypt the vector store, restrict access to it, log queries against it, and put it in your asset inventory and network diagram like any other sensitive repository. The practical consequence: your vector database must live inside the authorized boundary. For a DIB contractor, that means services like Azure AI Search running in Azure Government, or an equivalent 

inside AWS GovCloud, not a convenient SaaS vector service with unknown data residency and a consumer-grade DPA. 

Security Trimming: The Control That Makes or Breaks RAG 

Here is the most common RAG failure in real deployments. The team builds one big index over every repository the service account can reach, then puts a chat interface in front of it. The service account can read everything, so now every user can effectively read everything, one well-phrased question at a time. 

The fix is security trimming: retrieval results filtered to documents the asking user is authorized to open. Done properly, the index stores access control metadata alongside each chunk, the application resolves the user’s identity and group memberships at query time, and the search layer filters before anything reaches the model’s context window. Filtering after generation does not count. Once a passage enters the prompt, you have to assume its contents can surface in the output. Sensitivity labels compound the benefit here. If your chunks carry the labels of their source documents, you can enforce coarse rules on top of per-document ACLs, such as never retrieving Regulated content into sessions running outside the enclave, regardless of the individual user’s permissions. 

Every Arrow in the Diagram Is a Data Flow 

Draw the full RAG pipeline and follow the sensitive data: ingestion jobs read source repositories, an embedding model processes every chunk, the index stores everything, the orchestrator assembles prompts, the LLM endpoint receives them, and logs capture some or all of the above. 

Each arrow deserves the same questions you would ask about any system handling regulated data. Is the connection encrypted with approved cryptography? Does the component sit inside the authorized environment? Who operates it, and does that operator qualify as an external service provider you need to account for in your SSP and customer responsibility matrix? The embedding model is a frequently missed one: if you call a commercial embedding API from your pipeline, every chunk of every document transits that provider. In a CUI context, that provider needs to meet the same FedRAMP expectations as any other cloud service handling covered defense information. 

Prompt Logs: The Dataset Everyone Forgets 

Prompt and response logging deserves its own line item. Assembled prompts contain retrieved chunks, which means your observability platform may be quietly accumulating the most sensitive dataset in the company. Decide deliberately what gets logged, where logs live, how long they persist, and who can read them. 

Poisoning and Injection: RAG’s Two Native Attacks 

RAG also introduces integrity problems, not just confidentiality ones. If an attacker, or a careless user, can write to a repository your pipeline indexes, they can plant content that the model will faithfully retrieve and repeat. Worse, retrieved content can carry indirect prompt injection: instructions embedded in a document that attempt to steer the model’s behavior when they land in the context window. The OWASP Top 10 for LLM applications ranks prompt injection as the leading risk category for a reason. 

Mitigations are unglamorous and effective. Constrain what sources get indexed and who can write to them. Re-validate the corpus on a schedule. Treat retrieved text as untrusted input in your orchestration layer rather than as system instruction. And monitor for retrieval anomalies, such as a low-relevance document suddenly appearing in a high percentage of sessions. 

A Compliance-Shaped RAG Checklist 

Before a regulated RAG system goes live, you should be able to show: a data flow diagram covering ingestion, embedding, storage, retrieval, generation, and logging; every component placed inside an authorized boundary or formally accounted for as an external service; security trimming enforced at query time and tested with real user identities; the index, embeddings, and logs in your asset inventory with owners named; and an update and deletion process that reaches derived artifacts when source documents change or leave authorization. If you can produce that package, RAG becomes what it promised to be: the value of AI over your institutional knowledge, without your institutional knowledge escaping the boundary you built for it. 

Building RAG over CUI, patient data, or customer financial records? VSO designs and operates retrieval architectures inside Azure Government and AWS GovCloud enclaves, with security trimming and compliance-ready evidence built in. Schedule an architecture review with our team. Call VSO at (888) 805-0510 or email sales@vso-inc.com. 

FAQ