Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

• URL: http://arxiv.org/abs/2603.04428v1
• Date: Tue, 17 Feb 2026 05:46:20 GMT
• Title: Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
• Abstract summary: Multi-agent LLM systems on edge devices face a memory management problem.<n>On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16.<n>We address this by persisting each agent's KV cache to disk in 4-bit quantized format.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Related papers

• CacheBridge: Efficient Cross-Model KV Cache Transfer [12.058564378845373]
  Full-Head Mapping maps each target KV head from every source KV head in selected layers, making transfer quality sensitive to architectural differences.<n>We introduce CacheBridge, which co-designs architecture-indexed mapper support, attention-aligned calibration, and bounded mapper construction.<n>On Qwen3 $14mathrmBto32mathrmB$, it reduces mapper storage by $8times$, accelerates application by up to $3.0times$, matches fullhead with one tenth of the calibration data, and reduces 500-sequence construction from 92.63 to 8.
  arXiv  Detail & Related papers  (2026-09-01T08:20:35Z)
• RoPE-Aware Bit Allocation for KV-Cache Quantization [52.099459337231345]
  Block-GTQ is a bit allocator for key-cache quantization built on TurboQuant-MSE.<n>It better preserves RoPE query-key logits on a ten-model diagnostic panel.<n>It runs 1.34x faster than fp16 FlashAttention2 at 128K context.
  arXiv  Detail & Related papers  (2026-06-23T00:17:48Z)
• ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse [6.734363135861142]
  Prefix KV caching has become a key mechanism in LLM serving.<n>Current systems keep the KV cache in remote DRAM pools, increasing serving-cluster size and cost.<n>We propose ObjectCache, which co-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it.
  arXiv  Detail & Related papers  (2026-05-16T16:48:46Z)
• Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference [0.0]
  Key-value ( KV) cache memory management is the primary bottleneck in large-scale GPU inference serving.<n>Current systems suffer from three compounding inefficiencies.<n>We present a unified system that addresses all three problems.
  arXiv  Detail & Related papers  (2026-04-19T21:34:09Z)
• Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon [0.0]
  We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon.<n>It enables 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac.<n>Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation.
  arXiv  Detail & Related papers  (2026-04-18T10:39:28Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• EpiCache: Episodic KV Cache Management for Long Conversational Question Answering [15.288494370436469]
  We introduce EpiCache, a training-free KV cache management framework for long conversational question answering.<n>EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression.<n>Across three LongConvQA benchmarks, EpiCache improves accuracy by up to 40%, maintains near-full KV accuracy under 4-6x compression, and reduces latency/memory by up to 2.4x/3.5x.
  arXiv  Detail & Related papers  (2025-09-22T06:56:35Z)
• HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading [79.38548165722229]
  HEADINFER offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU.<n>We demonstrate HEADINFER maintains computational efficiency while significantly reducing memory footprint.
  arXiv  Detail & Related papers  (2025-02-18T06:26:05Z)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache [67.84112700032007]
  Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings.<n>In these scenarios, the Key-Value ( KV) cache is the primary bottleneck in terms of both GPU memory and latency.<n>We propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration.
  arXiv  Detail & Related papers  (2025-02-05T20:43:48Z)
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models [48.03117580340151]
  Key-Value ( KV) cache stores key-value states of previously generated tokens. The size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. We present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective.
  arXiv  Detail & Related papers  (2024-05-23T09:43:52Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)
• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving [31.766738294505767]
  CacheGen is a fast context-loading module for large language models. Uses a custom tensor encoder to encode a KV cache into compact bitstream representations. adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth.
  arXiv  Detail & Related papers  (2023-10-11T07:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.