Company with an on-site data centre
Case study: on-premises LLM inference, delivered with the hardware
For a company we sourced, built and commissioned an inference stack for its on-site data centre: three 2U servers with NVIDIA A100 GPUs, Kubernetes with GPU nodes, vLLM for inference and LiteLLM as the shared entry point for the company's applications. Plus high availability for the network underneath, with tested failovers for firewalls and switches.
- Frybad's role
- Sourcing, solution architecture, build, delivery and commissioning; then high availability for the network
- Hardware
- 3 × 2U rack servers (19"), memory and disks, NVIDIA A100 GPUs
- Stack
- Kubernetes with GPU nodes, vLLM, LiteLLM; firewalls and switches in a failover pair, OTP-based 2FA, monitoring