Serving engines
Open-source engines that serve open-weight models with an OpenAI-compatible API.
Changes fast: check the latest updates too. Reviewed 17 Oct 2026.
Key ideas
- vLLM: the default choice; PagedAttention, continuous batching, multi-LoRA, wide model support.
- SGLang: strong on structured generation and prefix sharing.
- TensorRT-LLM / NVIDIA NIM: maximum throughput on NVIDIA hardware.
- llama.cpp / Ollama: CPU and laptop inference with GGUF quantised models.
- Managed: Bedrock, SageMaker, Vertex, Azure AI Foundry, Together, Fireworks, Groq.
Common mistakes
- Self-hosting without a utilisation estimate; idle GPUs usually cost more than API calls at low volume.