On-prem AI is not a cost decision. It's a sovereignty decision.
The on-prem-versus-cloud conversation almost always becomes a cost conversation — API pricing, GPU infrastructure costs, break-even query volumes. What it almost never includes is the question that should come first: is cloud even an option? For a manufacturer under an aerospace supply chain contract with data sovereignty clauses, the answer is no. Not "no, it's expensive" — the contract prohibits sending certain document categories to third-party APIs, and that prohibition has no price. The same holds under GDPR or PDPA with personal data in queries, for institutions where model weight ownership is a regulatory requirement, and for healthcare organisations whose data classification makes cloud inference a compliance violation rather than a cost item. In each case the architecture isn't chosen. It's determined.
Sovereignty in AI deployment means control over three things, and each has a practical architectural consequence rather than a technical preference behind it.
Data sovereignty — where a query travels once submitted. A cloud API means it leaves your network, passes through third-party infrastructure, and returns a response. If the query carries proprietary process documentation or personal data, that information has crossed your legal boundary, and the question isn't whether that's acceptable on a cost basis. It's whether it's permitted at all.
Model weight sovereignty — the EU AI Act's high-risk provisions require operators to demonstrate weight ownership, training data lineage, and the ability to freeze and version a deployed model for audit. A proprietary API satisfies none of this: weights are inaccessible, training data went to a third-party server, and the model version is vendor-controlled, subject to change without notice. For a system that must be demonstrably compliant, these aren't inconveniences — they're disqualifying properties of the architecture.
Audit sovereignty — the ability to explain, on demand, exactly what the system did and why. In a cloud deployment you can log inputs and outputs; you cannot log the internal reasoning pathway or the exact weights active at the moment of a specific decision. For credit scoring, insurance risk assessment, or clinical decision support, the inability to reconstruct a specific inference for audit is not an operational inconvenience. It's a legal exposure.
Three forcing functions mandate on-prem architecture across the systems I've designed, and none of them are negotiable on cost grounds:
Contractual data classification. Supply chain contracts in aerospace, automotive, and defence routinely prohibit transmitting specified document categories to third-party systems — not a security preference, a contractual obligation with defined breach consequences. When the documents that need indexing fall into a prohibited category, cloud RAG isn't a weighed option. It's contractually excluded.
Regulatory data residency. GDPR Article 44 restricts personal data transfers to third countries without adequate protections; Singapore's PDPA Section 26 imposes similar restrictions. A cloud provider's local data centre doesn't resolve this — the data still passes through that provider's global infrastructure for processing. The only architecture that guarantees residency with certainty is one where inference runs on hardware you control, in a network boundary you control, with data that never leaves it.
Model weight ownership for high-risk AI. The EU AI Act classifies credit scoring, insurance, healthcare decision support, and employment management as high-risk, requiring documented evidence across five obligations — data governance, technical documentation, transparency, human oversight, accuracy verification. Every one requires access to the model weights. A proprietary API satisfies none of them, because the obligation isn't to use a good AI system. It's to be able to prove it.
On-prem is not the conservative choice in any of these contexts. It's the compliant one — and the cost conversation, when it happens at all, happens after the architecture has already been determined.
Further reading: European Parliament and Council (2024). Regulation (EU) 2024/1689 — Artificial Intelligence Act, Art. 9–17. · Personal Data Protection Commission Singapore (2012, amended 2020). PDPA, Section 26. · U.S. Department of State (2022). ITAR, 22 CFR Parts 120–130. · Nygard, M. (2011). Documenting Architecture Decisions. Cognitect.