MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration

• URL: http://arxiv.org/abs/2604.14889v1
• Date: Thu, 16 Apr 2026 11:29:40 GMT
• Title: MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
• Abstract summary: Chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning problems.<n>CoT reasoning faces scaling issues in terms of speed and memory usage.<n>We propose MemoSight (Memory-Foresight-based reasoning), a unified framework that integrates both context compression and multi-token prediction.
• Score: 43.3663361769054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning problems, as KV cache grows linearly with the number of generated tokens, CoT reasoning faces scaling issues in terms of speed and memory usage. In this work, we propose MemoSight (Memory-Foresight-based reasoning), a unified framework that integrates both context compression and multi-token prediction to mitigate the efficiency issues while maintaining CoT reasoning performance. Our framework adopts the same minimalist design for both context compression and multi-token prediction via special tokens and their corresponding position layout tailored to each token type. Comprehensive experiments on four reasoning benchmarks demonstrate that MemoSight reduces the KV cache footprint by up to 66% and accelerates inference by 1.56x, while outperforming existing CoT compression methods.

Related papers

• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference [20.58039191111583]
  Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation.<n>The resulting key-value ( KV) cache grows linearly with sequence length and creates severe memory bottlenecks.<n>We propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster.
  arXiv  Detail & Related papers  (2026-09-04T10:23:16Z)
• Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning [29.263926212060014]
  Large language models often rely on Chain-of-Thought (CoT) reasoning to solve complex tasks.<n>CoT compression shortens generation, yet aggressive compression may disrupt logical coherence and degrade performance.<n>We formalize this trade-off as the Context-Generation Substitution Law, where explicit reasoning context substitutes for part of decode-time generation.
  arXiv  Detail & Related papers  (2026-08-21T16:22:36Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoning [29.057579417751203]
  Chain-of-thought (CoT) prompting improves LLM reasoning but incurs high latency and memory cost due to verbose traces.<n>We propose textbfCtrlCoT, a dual-granularity CoT compression framework that harmonizes semantic abstraction and token-level pruning.
  arXiv  Detail & Related papers  (2026-01-28T10:38:49Z)
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs [57.217593337454026]
  TokenSqueeze is a novel Long2Short method that condenses reasoning paths while preserving performance and relying exclusively on self-generated data.<n>We show that TokenSqueeze reduces token usage while maintaining accuracy on the MATH500 benchmark.
  arXiv  Detail & Related papers  (2025-11-17T10:38:56Z)
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models [13.284627477293322]
  ThinKV is a thought-adaptive KV cache compression framework.<n>It applies a hybrid quantization-eviction strategy, assigning token precision by thought importance.<n>Experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache.
  arXiv  Detail & Related papers  (2025-10-01T04:09:02Z)
• Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains [24.805434364781306]
  We introduce Compressed Latent Reasoning (CoLaR), a novel framework that dynamically compresses reasoning processes in latent space.<n>CoLaR achieves 14.1% higher accuracy than latent-based baseline methods at comparable compression ratios.<n>Our RL-enhanced CoLaR demonstrates performance gains of up to 5.4% while dramatically reducing latent reasoning chain length by 82.8%.
  arXiv  Detail & Related papers  (2025-05-22T11:40:26Z)
• LightThinker: Thinking Step-by-Step Compression [74.34839026338342]
  We propose LightThinker, a method that enables large language models to dynamically compress intermediate thoughts during reasoning.<n>Inspired by human cognitive processes, LightThinker compresses thought steps into compact representations and discards the original reasoning chains.<n>Experiments show that LightThinker reduces peak memory usage and inference time, while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-02-21T16:57:22Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.