Company with an on-site data centre

LLM inference in an on-site data centre: hardware, architecture and operations from one source

Frybad's role
Sourcing, solution architecture, build, delivery and commissioning; then high availability for the network
Hardware
3 × 2U rack servers (19"), memory and disks, NVIDIA A100 GPUs
Stack
Kubernetes with GPU nodes, vLLM, LiteLLM; firewalls and switches in a failover pair, OTP-based 2FA, monitoring

Starting point

The customer wanted language models in its on-site data centre: for in-house applications, behind an interface that works like the well-known LLM APIs but runs on its own hardware. There was no GPU system and no platform for it. They were looking for one partner to take on sourcing, architecture, build and commissioning together.

After commissioning came a second engagement: the network the new platform hangs on had no redundancy. If the firewall or a switch failed, everything stopped.

What we did

Part 1: inference stack

  • Sourcing. Three 2U rack servers, plus memory, disks and NVIDIA A100 GPUs. We sized the configuration for the planned models and the space available in the customer’s rack.
  • Platform. A Kubernetes cluster with GPU nodes. GPUs are scheduled as a cluster resource, so models are rolled out, scaled and updated like any other workload.
  • Inference. vLLM serves the models and exposes an OpenAI-compatible API. It manages GPU memory through PagedAttention.
  • Company-wide proxy. LiteLLM sits in front of the models as the central entry point: one endpoint for all applications, keys per team, routing to the right model, usage visible per key. Applications talk to the proxy, not to individual servers.
  • Delivery and commissioning. The hardware was brought into the customer’s data centre, installed, brought online and handed over together with the platform.

Part 2: high availability for the network

  • Sourcing. Firewalls with pro-rated licences (aligned to the existing term), switches, and the licences for two-factor authentication (OTP) on the firewalls.
  • Configuration. The existing switch configuration was reviewed and migrated to the new devices.
  • Failover tests. For firewalls and switches, the failure of one device at a time was rehearsed and the switchover measured.
  • Hardening and monitoring. The firewalls are secured with 2FA; firewalls and switches are monitored.

What this means for your project

If you want to run language models or other GPU workloads, on hardware in your data centre or on our hardware in Munich, start with GPU servers & hosting in Germany. The platform underneath is described under Kubernetes platforms, built and run.

Result

Example from a client project: first the inference stack went into service, then the network failover tests ran.

The inference stack runs in the customer's data centre: Kubernetes with GPU nodes, vLLM, LiteLLM as the company-wide proxy. Hardware, architecture and solution came from one source.

The network stack is redundant: firewalls and switches paired, failover tested for both, firewall access with 2FA, both monitored.

Not included: model names, user numbers or throughput.

What we would do differently today

Network first. Redundancy for firewalls and switches came as the second engagement. Today we would plan it before commissioning the platform: an inference stack is only as available as the network it hangs on.

Failover tests as a runbook with a way back. Every test belongs in a maintenance window with documented steps and a described rollback, so it can be repeated, also without us.

Align licence terms. Pro-rated licences are cheaper to buy but create several renewal dates. One shared date is easier to run.

Similar situation?

Tell us which models and which data should stay in-house. We will tell you what hardware and platform that takes.