Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

Why it matters

Buoyant, the company behind the Linkerd service mesh, has released Kinference, a permissively licensed open source project that adds an inference control plane on top of Kubernetes and vLLM. Kinference sits in front of a fleet of vLLM workers and decides where each request is sent, combining routing policies, token rate limits, KV-cache-aware load balancing, and per-tenant metrics in a single layer. Policies are declarative rules that can match on tenant, requested model, GPU pool state, and semantic tags, enabling patterns such as steering low-complexity prompts to cheaper models, bursting to cloud providers, keeping tagged sensitive traffic local, and publishing a stable model alias backed by several models. Most rules optimize predicted time-to-first-token using cache overlap and queue length. Buoyant's self-reported benchmarks, against plain Kubernetes Service routing rather than another cache-aware router, show large median time-to-first-token improvements on shared-prefix traffic and no median gain on prompts without shared prefixes. The project is still alpha, and features including rate limiting and Anthropic API support are not yet shipped.