LLMKube

LLMKube is a Kubernetes operator that simplifies deploying and scaling large language models on your own hardware. It supports NVIDIA, Apple Silicon, and AMD GPUs, with pluggable runtimes including vLLM, llama.cpp, and TGI. The operator handles model management, GPU allocation, health monitoring, and exposes OpenAI-compatible APIs, turning complex infrastructure into a two-line YAML configuration.

Designed for teams needing data privacy, cost control, or air-gapped compliance, LLMKube addresses the scaling challenges of local LLM deployment. It includes ModelRouter for policy-aware routing between local and external providers, HPA autoscaling based on inference metrics, and Foreman, an agentic system that dispatches coding agents across heterogeneous fleets. The project is built with Kubernetes-native orchestration and is actively developed as open-source software.