From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
arXiv:2607.02574v1 Announce Type: cross Abstract: The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes -- local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current
The rapid growth and increasing scale of Large Language Models (LLMs) necessitate more efficient memory management solutions for their serving infrastructure to meet demand and control costs.
Efficient KV cache management is critical for scaling LLM inference, reducing operational costs, and optimizing performance, directly impacting the economic viability and widespread adoption of advanced AI models.
This classification and survey of KV cache management systems provide a clearer understanding of optimization pathways, leading to more robust and cost-effective distributed LLM serving architectures.
- · Cloud providers
- · AI model developers
- · LLM-reliant enterprises
- · Semiconductor manufacturers (memory)
- · Companies with inefficient LLM serving infrastructure
- · Competitors using less optimized memory solutions
Improved efficiency in LLM serving will lead to lower inference costs and higher throughput.
More affordable and scalable LLM access will accelerate the development and deployment of new AI applications and services.
The commoditization of LLM serving could shift value creation further towards foundational model development or application-layer innovation.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI