03 / 07 · Inference runtime

Choose an LLM inference runtime and serving framework

LLM Inference Engines and Serving Frameworks

Key concepts

An inference engine or runtime executes the model. A serving framework handles requests, scheduling, concurrency, and service endpoints. Tools can span both layers.
llama.cpp provides local inference and an HTTP server. Ollama packages model management and execution.
Evaluate SGLang and vLLM for serving. Include Strata when testing supported MoE and RAM/VRAM layouts. Check GPU backends, formats, quantization, versions, and APIs.

Colleagues need local coding and document tools. What should the inference software optimize first?

Fast onboarding · Ollama
Fits repeatable onboarding and common client integrations.
Tradeoffs · Convenience still requires feature checks, authentication, and monitoring for shared service.
Verify · Rebuild from a clean environment and verify local execution, API, and real tool calls.
Control residency · llama.cpp / Strata
Fits explicit tuning of backends, context, quantization, or supported hybrid MoE.
Tradeoffs · More controls require pinned versions, launch configurations, and regression records.
Verify · Save reproducible commands and measure quality, memory peaks, and latency.
Shared serving · vLLM / SGLang
Fits concurrent requests and service-level throughput targets.
Tradeoffs · Requires supported models and accelerators, deployment work, and capacity planning.
Verify · Load-test mixed request lengths and inspect queueing, first-token, and completion latency.

Record the model, runtime version, and launch method, then reproduce one local inference run.

MODEL MANAGEMENT · AI-generated concept illustration. Verify actual specifications in product documentation.
RUNTIME TUNING · AI-generated concept illustration. Verify actual specifications in product documentation.
CONCURRENT SERVING · AI-generated concept illustration. Verify actual specifications in product documentation.