A Universal Context-Reuse Layer for Cross-Model KV Sharing

• URL: http://arxiv.org/abs/2608.30963v1
• Date: Mon, 31 Aug 2026 15:28:17 GMT
• Title: A Universal Context-Reuse Layer for Cross-Model KV Sharing
• Abstract summary: We study emphcross-model KV sharing, which translates the KV state produced by a source model into a representation that can be consumed by a different target model.<n>For Qwen2.5-7B $rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48%, a gain of 6.89 percentage points over the native 1.5B baseline.<n>For the cross-family Qwen2.5-1.5B $rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by
• Score: 7.914652203965336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B $\rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59\% to 34.48\%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B $\rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05\% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B $\rightarrow$ Qwen2.5-7B setting, cross-family handoff achieves 44.0\% accuracy compared with 45.7\% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.

Related papers

• Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse [3.9548951874725433]
  We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill.<n>We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension.<n>This linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply.
  arXiv  Detail & Related papers  (2026-08-04T16:26:47Z)
• Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models [2.1665689529884697]
  We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM.<n>B1ade-embed, a 335M parameter retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training.<n>B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points.
  arXiv  Detail & Related papers  (2026-07-29T22:41:25Z)
• Variable-Width Transformers [57.07167443557886]
  We investigate nonuniform capacity allocation across network depth by proposing a $times$-shaped > former architecture.<n>Our design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-free residual resizing mechanism.<n>By reducing the average layer width, this architecture also requires fewer overall FLOPs and smaller KV cache memory and I/O cost.
  arXiv  Detail & Related papers  (2026-06-16T17:59:03Z)
• VisualClaw: A Real-Time, Personalized Agent for the Physical World [76.89211120690028]
  We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
  arXiv  Detail & Related papers  (2026-06-15T06:58:22Z)
• Latent-Condensed Transformer for Efficient Long Context Modeling [60.72493959155964]
  We propose Latent-Condensed Attention (LCA) for large language models.<n>LCA condenses context within MLA's latent space, where representation is disentangled into semantic latent vectors and positional keys.<n>LCA achieves up to 2.5$times$ prefilling speedup and 90% KV cache reduction at 128K context.
  arXiv  Detail & Related papers  (2026-04-14T08:40:31Z)
• Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment [0.05586191108738562]
  Small language models (SLMs) have sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification.<n>Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus.<n>Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm.
  arXiv  Detail & Related papers  (2026-03-26T15:57:46Z)
• Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs [3.18088418556364]
  We provide the first theoretical justification for layer-adaptive merging.<n>We propose textbfFIM-Merging, which computes diagonal FIM using only random token inputs.<n>Our framework also provides a unified theoretical explanation for why existing layer-adaptive methods such as ACM empirically outperform uniform merging.
  arXiv  Detail & Related papers  (2026-03-23T08:45:37Z)
• Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection [15.230462656535034]
  Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = dmodel$)<n>Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary.<n>We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $BigO(log N) dimensions to distinguish among $N relevant patterns.
  arXiv  Detail & Related papers  (2026-02-16T23:45:39Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference [0.0]
  We present Entropy Adaptive Decoding (EAD), a novel approach for efficient language model inference.<n>EAD switches between different-sized models based on prediction uncertainty.<n>We show remarkable efficiency gains across different model families.
  arXiv  Detail & Related papers  (2025-02-05T22:15:21Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)
• Training Compute-Optimal Large Language Models [54.00424650998489]
  We train language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. For compute-optimal training, the model size and the number of training tokens should be scaled equally. chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)
  arXiv  Detail & Related papers  (2022-03-29T13:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.