The real hardware math behind “on-premise”

“On-premise supported” shows up on nearly every vendor’s feature list as a single checkbox. It isn’t one number. Whether that means a $25,000 server under a desk or a $300,000 rack in a purpose-built room (GPU hardware is quoted in USD almost everywhere) depends entirely on three choices that rarely make it into the sales deck: which model, at what precision, and how many GPUs it takes to hold it in memory.

What “on-premise” actually requires

Most of the strongest open-weight models shipping in 2026, Llama 4 Maverick among them, are mixture-of-experts: 400 billion total parameters, but only 17 billion active for any given token, spread across 128 experts. The active-parameter number is what determines inference speed. It is not what determines VRAM. Every expert has to sit in memory in case the router picks it, so a 17B-active model still needs enough GPU memory for all 400B parameters, roughly 800GB at full FP16 precision, before a single token is generated.

What quantization saves, and what it costs

4-bit quantization (INT4/Q4) cuts that same model to roughly 200 to 224GB, within reach of four 80GB datacenter GPUs instead of eight. That’s a genuine, usable trade, most production deployments run quantized. It is not free: push quantization further, some setups fit an even larger model into two 48GB cards at under 2 bits, and the quality cost stops being negligible. Profile the real task at the real quantization level before it goes into a contract.

Often a smaller model is the right answer

Not every deployment needs a 400B-parameter model at all. A strong dense 27B-class open model runs comfortably on a single 80GB GPU, no cluster, no distributed inference, a fraction of the hardware bill of the frontier tier above. For most Mittelstand-scale document and query workloads, that’s the sensible starting point rather than a fallback for when the big model turns out too expensive to run.

Sovereignty doesn’t require an air gap

The EU AI Act reaches full application on 2 August 2026, and several of the use cases German Mittelstand companies run through this kind of system, credit scoring among them, are classified high-risk, which triggers real compliance obligations. None of that is a mandate to physically own the GPUs. What it does require, especially layered with DORA for regulated financial entities, is audit rights, control over your own data, and managed concentration risk, not a specific rack in a specific room. A German-hosted private cloud with a signed data processing agreement satisfies that for most sectors. True air-gapped on-premise earns its cost where a regulator specifically demands it, not by default.

A buyer’s checklist

Before an “on-premise supported” line item goes into any contract, four questions settle whether it means what it sounds like: which precision the demo was running at, how many GPUs does that precision require at the model size you were shown, who operates and patches that hardware once it’s yours, and what happens to answer quality if the production deployment turns out to need a smaller model than the pitch.