Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning

• URL: http://arxiv.org/abs/2608.21265v2
• Date: Wed, 26 Aug 2026 12:50:17 GMT
• Title: Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning
• Abstract summary: Large language models often rely on Chain-of-Thought (CoT) reasoning to solve complex tasks.<n>CoT compression shortens generation, yet aggressive compression may disrupt logical coherence and degrade performance.<n>We formalize this trade-off as the Context-Generation Substitution Law, where explicit reasoning context substitutes for part of decode-time generation.
• Score: 29.263926212060014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models often rely on Chain-of-Thought (CoT) reasoning to solve complex tasks, but verbose reasoning traces introduce substantial inference overhead. CoT compression shortens generation, yet aggressive compression may disrupt logical coherence and degrade performance. We formalize this trade-off as the Context-Generation Substitution Law, where explicit reasoning context substitutes for part of decode-time generation. Based on this principle, we propose Memory-Augmented Compression, a training-free framework that constructs reusable reasoning memories from historical traces and retrieves them as prefill-side scaffolds. Rather than using raw demonstrations, these memories summarize reusable reasoning patterns, key constraints, and critical operations to compensate for information lost during compression. Experiments show that Memory consistently improves prompt-based Chain-of-Draft (CoD) compression across mathematical reasoning, complex reasoning, and science question answering tasks, yielding accuracy gains of 21.4, 28.0, 29.5, and 6.61 points over CoD on GSM8K, MATH, BBH, and MMLU-Sci, while achieving a 1.14-1.49x latency speedup latency speedup over standard CoT. Memory is also compatible with token-level, reasoning-trace-level, and inference-state compression mechanisms.

Related papers

• MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration [43.3663361769054]
  Chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning problems.<n>CoT reasoning faces scaling issues in terms of speed and memory usage.<n>We propose MemoSight (Memory-Foresight-based reasoning), a unified framework that integrates both context compression and multi-token prediction.
  arXiv  Detail & Related papers  (2026-04-16T11:29:40Z)
• LightThinker++: From Reasoning Compression to Memory Management [61.2260619973687]
  Large language models (LLMs) excel at complex reasoning, yet their efficiency is limited by the surging cognitive overhead of long thought traces.<n>We propose LightThinker, a method that enables LLMs to dynamically compress intermediate thoughts into compact semantic representations.<n>We evolve the framework into LightThinker++, introducing Explicit Adaptive Memory Management.
  arXiv  Detail & Related papers  (2026-04-04T10:46:09Z)
• Shorter Thoughts, Same Answers: Difficulty-Scaled Segment-Wise RL for CoT Compression [3.6889211112573985]
  Chain-of-thought (CoT) improves reasoning reliability but increases token cost.<n> naive RL-based compression can undesirably shorten the user-facing answer.<n>We propose Difficulty-Scaled Segment-Wise GRPO.
  arXiv  Detail & Related papers  (2026-03-08T11:49:24Z)
• Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning [47.87361916374891]
  We propose a framework for efficient long-context inference based on chunk-wise compression and selective memory recall.<n>The framework segments long inputs into chunks and encodes each chunk into compressed memory representations using a learned compressor.<n>It achieves up to a 2 times reduction in peak GPU memory usage and a 6 times inference speedup over MemAgent.
  arXiv  Detail & Related papers  (2026-02-09T08:33:11Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thought [49.203970812338916]
  Explicit reasoning chains introduce substantial computational redundancy.<n>Recent latent reasoning methods attempt to mitigate this by compressing reasoning processes into latent space.<n>We propose Rendered CoT-Guided variational Latent Reasoning (ReGuLaR)
  arXiv  Detail & Related papers  (2026-01-30T17:08:06Z)
• CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoning [29.057579417751203]
  Chain-of-thought (CoT) prompting improves LLM reasoning but incurs high latency and memory cost due to verbose traces.<n>We propose textbfCtrlCoT, a dual-granularity CoT compression framework that harmonizes semantic abstraction and token-level pruning.
  arXiv  Detail & Related papers  (2026-01-28T10:38:49Z)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancy [8.962703809086628]
  ThinkLess is an inference-efficient framework that terminates reasoning generation early and maintains output quality without modifying the model.<n>We show that ThinkLess achieves comparable accuracy to full-length Chain-of-Thought (CoT) decoding while greatly reducing decoding time and memory consumption.
  arXiv  Detail & Related papers  (2025-05-21T15:58:16Z)
• LightThinker: Thinking Step-by-Step Compression [74.34839026338342]
  We propose LightThinker, a method that enables large language models to dynamically compress intermediate thoughts during reasoning.<n>Inspired by human cognitive processes, LightThinker compresses thought steps into compact representations and discards the original reasoning chains.<n>Experiments show that LightThinker reduces peak memory usage and inference time, while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-02-21T16:57:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.