RAG is not a feature. It's a constraint decision.

When I started designing VaultRAG — a retrieval system for factory floor use, running entirely on-prem — the first question I was asked was which embedding model to use. Reasonable, well-informed, and not the right place to start. Before the embedding model mattered, I had to decide whether the index could ever leave the facility network. Before that, whether the system needed to answer questions about documents updated monthly or in real time. Before that, whether a technician standing next to a machine with an alarm going off could afford a four-second response, or whether four seconds was already too slow. Each of those was a constraint, and each constraint eliminated certain architectures before any model comparison was relevant. By the time embedding model selection came up, half the options were already off the table — not because they underperformed, but because the constraints had made them architecturally impossible.

Retrieval-Augmented Generation gets sold as a capability you bolt on — add a vector store, write some chunking logic, now your model knows about your documents. That framing is technically accurate and practically misleading, because RAG is a systems decision, not a component decision. Choosing it commits you to a position on four things simultaneously, whether you realised you were deciding them or not.

Freshness. RAG retrieves from an index, and an index is a snapshot. The moment a document changes, the index is stale until you re-embed it. For equipment manuals revised quarterly, manageable. For a system answering questions about today's machine status, a snapshot-based index is architecturally wrong — you've implicitly decided your answers can be as stale as your last ingestion run.

Latency budget. A RAG pipeline adds latency relative to a model call alone — embed the query, run similarity search, retrieve chunks, generate. For VaultRAG the target was under eight seconds from voice query to cited response, because a technician with an alarm going off cannot wait longer. That constraint shaped the model size, the vector store, the chunk size, the number of retrieved documents. The latency budget isn't a performance requirement optimised at the end. It's a constraint designed to from the start.

Data residency. Cloud RAG means documents leave your network — to an embedding API, a managed vector store, a cloud inference endpoint. For manufacturing firms in aerospace, automotive, and defence supply chains, that's often contractually prohibited, not from security anxiety but from legally binding data sovereignty requirements. This doesn't make cloud RAG wrong. It makes it unavailable, and the architecture has to be designed around what's actually permitted.

Ongoing cost structure. Cloud RAG prices on API calls, storage, and compute — negligible at low volume, compounding quickly at the scale of hundreds of machines and continuous queries through production hours. The cost is variable and scales with usage, meaning a successful deployment costs more than a quiet one. On-prem inference inverts that: higher upfront capital cost, near-zero marginal cost per query. For a five-year deployment, the total cost of ownership calculation is completely different.

None of these four decisions appear in the average RAG tutorial, because tutorials start at the embedding model — that's where the interesting technical comparisons live. But the teams that build RAG systems well understand these four constraints going in. The teams that spend six months patching a RAG system usually discover them the other way around — after the architecture has already committed to a position none of them actually chose.

Further reading: Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arXiv:2005.11401. · Thakur, N. et al. (2021). BEIR. NeurIPS. arXiv:2104.08663. · Hu, E. et al. (2022). LoRA. ICLR. arXiv:2106.09685. · Nygard, M. (2011). Documenting Architecture Decisions.