vLLM v0.34.0 adds 10 models and enables extensible KV cache by default
It adds ten model architectures, among them Mistral Large 4, MiniMax-M3.1-Flash, NVIDIA Nemotron-3.5-Transcribe, Baidu Unlimited-OCR-2 and Liquid AI's LFM3 family. The extensible KV cache is now the default on CUDA and ROCm: cache size is derived from the memory left over after warmup and CUDA graph capture rather than an estimate, which the project says gives most models a larger cache. Model Runner V1 is removed, the default block size moves from 16 to 64, and ROCm 7.2 builds are no longer published, so existing deployments may need configuration changes.
Why it matters
Sizing the KV cache from measured free memory instead of an estimate lets most self-hosted deployments hold more context and serve more concurrent requests, though the project notes per-token latency can rise on some models unless operators opt out.
Sources
- PrimaryvLLM
Written by AIToolMaps from the source above and checked against it sentence by sentence.