AWS adds KV cache-aware routing to SageMaker AI endpoints
AWS introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints. The router uses cache state reported by the model container to send each request to an instance holding the longest cached match for its prompt, weighing that match against how busy the instance is. If no useful match exists or candidates are busy, it falls back to load-aware routing. It targets LLM workloads with multi-turn chats, long shared prompts, or heavy load, and requires at least two instances. Teams enable it by adding an environment variable so the container publishes cache events and setting RoutingStrategy to KV_AWARE in the endpoint configuration; clients keep using the same APIs. AWS reports lower time to first token and higher throughput than random routing in its benchmarks.
Why it matters
Chat, RAG, and agent applications resend long shared prompts, so routing to instances that actually hold the cached prefix can avoid repeated computation without requiring client changes.
Sources
- PrimaryAWS Machine Learning
Written by AIToolMaps from the source above and checked against it sentence by sentence.