LRU is harder to beat than the KV-cache papers suggest
A technical analysis of LLM serving infrastructure reveals that the standard Least Recently Used (LRU) eviction policy for KV-caches is surprisingly effective for agentic workloads. The author demonstrates that common assumptions about session idling are often incorrect, making LRU difficult to outperform in practice.
Why it matters
Optimizing LLM serving infrastructure is essential for reducing the high costs and latency associated with deploying AI agents at scale.
I replayed 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests through a prefix-cache simulator, tried to beat the production baseline three different ways, and failed. The interesting part is why: under capacity pressure, most recomputation comes from tool-calling loops seconds apart, not from sessions idling past a TTL — and the TTL never fires at all.
Everything here reproduces from a cold checkout with make setup data repro .
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in