Buoyant Launches Kinference to Manage AI Model Traffic
Buoyant has launched Kinference, an open source tool that helps companies manage how AI models answer requests on Kubernetes servers. It lets teams set routing rules, spreads work based on cached memory, and tracks usage for each customer. In Buoyant's own test, the time before the first reply dropped by 95 percent.
Quick glossaryBuoyant, Kubernetes, inference, TTFT
- Buoyant
- Buoyant is a US software company best known for Linkerd, a tool that helps cloud applications communicate reliably with each other.
- Kubernetes
- A system that automatically runs and organises many software programs across a group of computers, usually inside large companies.
- inference
- The step where a trained AI model takes a question or input and produces an answer.
- TTFT
- Time to first token, meaning how long an AI takes before it starts showing its first word of a reply.
Burning Potato score: 64%