Guide
On-premises LLM inference: a self-hosted stack with vLLM and LiteLLM on Kubernetes
A self-hosted inference stack has four layers: GPU servers, a Kubernetes cluster with GPU nodes, vLLM as the engine that serves the models, and LiteLLM as the proxy every application in the company talks to, with its own keys and budgets. This article covers the layers, the hardware and model decisions, and how to keep GPU memory under control.
It is written for companies that want to use language models but cannot let their prompts leave the building.
1. When a stack of your own makes sense
Most companies start with an external API, and that is the right call as long as the data is suitable. It stops being right the moment contracts, customer records, source code or internal documents are meant to go into prompts and your legal team or a customer says no. Then there are two routes: a European provider with a contract that fits, or your own hardware.
Your own stack pays off if at least one of these holds: the data must not leave the premises; usage is high and predictable, so a hardware investment competes with a running API bill; or you need models and settings that no provider offers. For one customer we sourced, built and brought online exactly such a stack in their data centre: three 2U servers with NVIDIA A100 GPUs, Kubernetes with GPU nodes, vLLM, and LiteLLM as the company-wide proxy. The decisions that came up are the sections below.
2. The four layers
| Layer | Job | Tool |
|---|---|---|
| Hardware | GPU memory and compute | Servers with data-centre GPUs (NVIDIA A100 or H100 class), enough RAM, NVMe for model weights |
| Orchestration | Hand out GPUs as a resource, restart services, roll out updates | Kubernetes with the NVIDIA GPU Operator |
| Inference engine | Load the model, batch requests, generate tokens | vLLM, one service per model |
| Access layer | One API for every application, keys, budgets, logs, fallbacks | LiteLLM proxy |
Each layer can be swapped without touching the others. That is the reason not to skip the access layer: applications talk to LiteLLM, never to vLLM. Swap a model and nothing changes for the application.
3. Hardware: GPU memory is the currency
“Which GPU?” is really “how much GPU memory?”. A model has to fit entirely into the GPU’s memory, plus room for the KV cache, which grows with every concurrent request and every token of context.
Rules of thumb for planning, to be re-measured with your own models:
- Weights: parameter count times bytes per parameter. A 7-billion-parameter model needs about 14 GB at 16-bit precision and about 3.5 GB with 4-bit quantisation.
- KV cache: grows linearly with context length and the number of concurrent requests. It is why a GPU the model “just about” fits on is full the moment production traffic arrives.
- Headroom: for the CUDA context and anything else sharing the card.
Previous-generation data-centre GPUs are often the more economical choice for inference than the newest ones: memory is the same bottleneck, and the price per gigabyte is far lower. More important than the generation is that the server, power feed, cooling and rack space match the card. If you source the hardware yourself, settle these before ordering: form factor (2U with room for the cards), power supplies, airflow, RAM at least the size of GPU memory, NVMe for the weights.
4. vLLM: the engine and its four most important knobs
vLLM serves a model behind an OpenAI-compatible HTTP interface and batches requests so the GPU stays busy. Two properties make it the operator’s choice: PagedAttention manages the KV cache in pages, like an operating system’s virtual memory, and so avoids fragmentation; continuous batching admits new requests into a running batch instead of waiting for it to finish.
From our own practice on our GPU servers, these are the four knobs that decide stability and throughput:
- Model choice. Not every task needs a 70-billion-parameter model. Classification, extraction and simple RAG answers run faster on smaller models with a fraction of the memory.
- Quantisation. 4-bit weights instead of 16-bit save roughly three quarters of the memory for weights, usually at a barely measurable quality cost. Measure that cost on your own tasks, not on benchmarks from the internet.
gpu_memory_utilization. By default vLLM reserves most of the GPU memory. If several services share a card, lower the value and leave room.max_model_len. The maximum context length decides how much KV cache vLLM sets aside per request. A realistic value for your application, rather than the model’s maximum, stops memory being reserved for context nobody uses.
One Deployment per model, with nvidia.com/gpu as the resource:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-chat
spec:
replicas: 1
selector:
matchLabels:
app: vllm-chat
template:
metadata:
labels:
app: vllm-chat
spec:
containers:
- name: vllm
image: vllm/vllm-openai # pin the tag per release
args:
- --model=/models/chat-model
- --quantization=awq
- --gpu-memory-utilization=0.85
- --max-model-len=16384
- --served-model-name=chat
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: models
mountPath: /models
volumes:
- name: models
persistentVolumeClaim:
claimName: model-weights
Model weights live on a volume, not in the image: images carrying ten gigabytes of weights make every rollout slow and every registry full.
5. LiteLLM: the proxy the company talks to
LiteLLM puts an OpenAI-compatible API in front of any number of backends. For a company it is the layer that turns “one server with one model” into a service:
- Virtual keys per team or application, each with a budget and a rate limit. Nobody gets direct access to the engine.
- Model aliases: the application asks for
chat; the proxy decides whether that is a local model today or, where permitted, an external provider. Model swaps without code changes. - Fallbacks and routing: if the local model is saturated, the proxy can hand over to a second instance or, for non-sensitive data, to an external provider. Whether that is allowed is decided per key in configuration, not by the application.
- Logging and cost: who used how many tokens. That is the basis for internal charge-back and for the question whether the hardware is busy enough.
An excerpt of the proxy configuration:
model_list:
- model_name: chat
litellm_params:
model: openai/chat
api_base: http://vllm-chat.inference.svc:8000/v1
api_key: 'none'
- model_name: extract
litellm_params:
model: openai/extract
api_base: http://vllm-extract.inference.svc:8000/v1
api_key: 'none'
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY # from a Kubernetes Secret
litellm_settings:
drop_params: true
The proxy needs a database for keys and usage (Postgres) and belongs behind the same ingress and authentication as your other internal services.
6. Operations: what comes after the first start
Driver updates. The NVIDIA GPU Operator installs the driver and container toolkit on GPU nodes. Unattended operating-system updates on those nodes must not swap the kernel without coordination; a new kernel without a matching driver module means a GPU the cluster no longer sees. Plan node updates as maintenance windows with drain and test.
Monitoring. Three metrics are enough to begin with: GPU memory utilisation, vLLM’s request queue length, and tokens per second. If the queue grows while throughput stays flat, the card is at its limit.
Model changes. A new model is a new Deployment next to the old one; the alias in the proxy is switched once tests pass. Rollback is switching the alias back.
Security. The engine is never reachable from outside, only the proxy. Weights come from a verified source and are stored with a checksum. Prompts and responses are logged only as long as the purpose requires; that belongs in the service’s data-protection documentation.
7. What it costs, and how to work it out
Do not quote a number before you have measured. The arithmetic is simple in both directions:
- External: monthly tokens times price per token, from the last few months of invoices or a pilot phase.
- Own: hardware depreciated over three to five years, plus power (server draw times hours times tariff), plus rack and network costs, plus the operating hours of your team or a service provider.
The curve tips towards own hardware at constantly high usage. At sporadic usage the external API stays cheaper, and the case for a stack of your own is then data protection alone, which can be an entirely sufficient case.
8. Planning checklist
Requirements:
- What data goes into prompts? May it leave the premises?
- Which tasks: chat, extraction, classification, code? One model candidate per task.
- Expected concurrent users and typical context length.
Hardware:
- GPU memory per model calculated (weights plus KV cache plus headroom).
- Server form factor, power supplies, cooling, rack space and power feed settled.
- NVMe for weights; RAM at least the size of GPU memory.
Platform:
- Kubernetes with the GPU Operator; GPU nodes excluded from unattended kernel updates.
- One vLLM Deployment per model, weights on volumes.
gpu_memory_utilizationandmax_model_lenset deliberately per service.
Access:
- LiteLLM proxy with virtual keys, budgets and aliases; database for usage.
- Engine unreachable from outside; proxy behind ingress and authentication.
- Logging rules for prompts and responses documented.
Operations:
- Metrics: GPU memory, queue length, tokens per second.
- Procedure for model swap and rollback via the alias.
- Cost comparison external vs. own with measured tokens.
9. Frequently asked questions
Is a consumer GPU enough to start with? For experiments, yes. For operations it lacks the memory, ECC protection and data-centre rating. Anyone building applications on it migrates later anyway.
Ollama instead of vLLM? Ollama is made for a single workstation and is easy to start. vLLM is built for many concurrent requests and uses the GPU far better through batching. Once several applications or users send requests at the same time, vLLM is usually the better fit.
Can I have both, local and external? Yes, that is what the proxy is for. Per key you define which models are allowed. Non-sensitive applications may go external, sensitive ones stay local.
How big does the team have to be? After the build it is the same operational work as any other Kubernetes service: updates, monitoring, the occasional model swap. If you already run Kubernetes, you do not need a new role for it.
Where this leads
We run our own hardware in a data centre in Munich, and our own inference runs there too. How we provide GPU capacity or build a stack like this in your data centre is described under GPU servers & hosting in Germany.
Example from a project: On-premises LLM inference: hardware and stack for a data centre.
German version: LLM-Inferenz on-premises mit vLLM und LiteLLM.
Sources
All read on 20 Oct 2026.
- vLLM documentation: Engine arguments, quantization, PagedAttention.
- LiteLLM documentation: Proxy server, virtual keys, budgets, fallbacks.
- NVIDIA: GPU Operator documentation.
The matching serviceGPU servers & hosting in Germany