vLLM 0.34.0 removes Model Runner V1 and sizes its KV cache from free GPU memory

TL;DR

  • The vLLM project published v0.34.0 on 16 November, with 10 new models, Mistral Large 4 among them.
  • The extensible KV cache is on by default, gpu_memory_utilization defaults to 1.0, and the default block size is 64.
  • Model Runner V1 is gone, so VLLM_USE_V2_MODEL_RUNNER=0 now fails at startup.

Read the full story

Sign in with your email to read AI News. It’s free.

Share this story

Explain like I’m 15