ProductsNov 17, 2026, 4:11 PM EST

AWS adds KV cache-aware routing to SageMaker AI endpoints

AWS introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints. The router uses cache state reported by the model container to send each request to an instance holding the longest cached match for its prompt, weighing that match against how busy the instance is. If no useful match exists or candidates are busy, it falls back to load-aware routing. It targets LLM workloads with multi-turn chats, long shared prompts, or heavy load, and requires at least two instances. Teams enable it by adding an environment variable so the container publishes cache events and setting RoutingStrategy to KV_AWARE in the endpoint configuration; clients keep using the same APIs. AWS reports lower time to first token and higher throughput than random routing in its benchmarks.

Why it matters

Chat, RAG, and agent applications resend long shared prompts, so routing to instances that actually hold the cached prefix can avoid repeated computation without requiring client changes.

Sources

Written by AIToolMaps from the source above and checked against it sentence by sentence.

Comments

No comments yet