EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation

• URL: http://arxiv.org/abs/2506.04205v1
• Date: Wed, 04 Jun 2025 17:49:10 GMT
• Title: EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation
• Abstract summary: We study the problem of CoT condensation for resource-efficient reasoning training.<n>We propose an Edge-Preserving Condensation method, EPiC, which selectively retains only the initial and final segments of each CoT trace.
• Score: 37.6583581020347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have shown remarkable reasoning capabilities when trained with chain-of-thought (CoT) supervision. However, the long and verbose CoT traces, especially those distilled from large reasoning models (LRMs) such as DeepSeek-R1, significantly increase training costs during the distillation process, where a non-reasoning base model is taught to replicate the reasoning behavior of an LRM. In this work, we study the problem of CoT condensation for resource-efficient reasoning training, aimed at pruning intermediate reasoning steps (i.e., thoughts) in CoT traces, enabling supervised model training on length-reduced CoT data while preserving both answer accuracy and the model's ability to generate coherent reasoning. Our rationale is that CoT traces typically follow a three-stage structure: problem understanding, exploration, and solution convergence. Through empirical analysis, we find that retaining the structure of the reasoning trace, especially the early stage of problem understanding (rich in reflective cues) and the final stage of solution convergence, is sufficient to achieve lossless reasoning supervision. To this end, we propose an Edge-Preserving Condensation method, EPiC, which selectively retains only the initial and final segments of each CoT trace while discarding the middle portion. This design draws an analogy to preserving the "edge" of a reasoning trajectory, capturing both the initial problem framing and the final answer synthesis, to maintain logical continuity. Experiments across multiple model families (Qwen and LLaMA) and benchmarks show that EPiC reduces training time by over 34% while achieving lossless reasoning accuracy on MATH500, comparable to full CoT supervision. To the best of our knowledge, this is the first study to explore thought-level CoT condensation for efficient reasoning model distillation.

Related papers

• WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models [51.88075626243057]
  We propose WhisperRec, an efficient latent reasoning framework for foundation recommendation models (FRMs)<n>WhisperRec compresses teacher-generated Chain-of-Thought (CoT) into learnable latent reasoning tokens.<n>We show that WhisperRec consistently outperforms explicit-CoT methods and conventional baselines.
  arXiv  Detail & Related papers  (2026-07-29T08:48:35Z)
• SuCo: Sufficiency-guided Continuous Adaptive Reasoning [59.17222918775686]
  Large Reasoning Models (LRMs) often generate excessively long Chain-of-Thoughts (CoT)<n>Existing efforts to mitigate this inefficiency typically rely on discrete reasoning modes or fixed budget tiers.<n>We introduce Minimal Sufficient CoT (MSC), defined as the shortest prefix of a CoT trajectory which is adequate for producing the correct answer.
  arXiv  Detail & Related papers  (2026-06-16T08:52:15Z)
• LoRi: Low-Rank Distillation for Implicit Reasoning [7.490026305145753]
  Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting.<n>We propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace.
  arXiv  Detail & Related papers  (2026-06-03T18:05:50Z)
• CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning [25.840307083323456]
  Long Chain-of-Thought (CoT) reasoning is pivotal for the success of recent reasoning models but suffers from high computational overhead and latency.<n>This paper presents textbfCompressing textbfRedundancy in Chain-of-Thought via textbfIntrinsic textbfSaliency textbfPruning (textbfCRISP), a framework that compresses CoT by exploiting the model's intrinsic saliency.
  arXiv  Detail & Related papers  (2026-04-19T07:26:55Z)
• Constraint-Rectified Training for Efficient Chain-of-Thought [60.52883907721588]
  Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs)<n>While longer reasoning traces can improve answer quality and unlock abilities such as self-correction, they also incur high inference costs and often introduce redundant steps, known as overthinking.<n>Recent research seeks to develop efficient reasoning strategies that balance reasoning length and accuracy.
  arXiv  Detail & Related papers  (2026-02-13T02:13:45Z)
• Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks [37.23113613155819]
  Latent Chain-of-Thought (Latent-CoT) aims to enable step-by-step computation without emitting long rationales.<n>We study CODI, a continuous-thought teacher-student distillation model, on strictly sequential-it tasks.
  arXiv  Detail & Related papers  (2026-01-31T01:48:23Z)
• Chain Of Thought Compression: A Theoritical Analysis [24.613200477865572]
  Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models.<n>CoT incurs prohibitive computational costs due to generation of extra tokens.<n>Recent studies show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative.
  arXiv  Detail & Related papers  (2026-01-29T11:42:03Z)
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit [114.83867400179354]
  Overthinking can degrade overall performance of large language models.<n>We categorize reasoning into three stages: insufficient exploration stage, compensatory reasoning stage, and reasoning convergence stage.<n>We develop a lightweight thresholding strategy based on rules to improve reasoning accuracy.
  arXiv  Detail & Related papers  (2025-08-25T03:17:17Z)
• Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens [23.326813303795692]
  Chain-of-Thought (CoT) prompting has been shown to improve Large Language Model (LLM) performance on various tasks.<n>However, some initial findings suggest that CoT reasoning may be more superficial than it appears.
  arXiv  Detail & Related papers  (2025-08-02T04:37:28Z)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought [37.53215651690168]
  Chain of Thought (CoT) prompting improves the reasoning performance of large language models (LLMs) by encouraging step by step thinking.<n>While promising, CoT-based approaches often require costly pretraining and lack a principled framework for how reasoning should evolve.<n>We propose SCOUT, a lightweight fine tuning framework that enables Flow CoT style reasoning without the need for pretraining.
  arXiv  Detail & Related papers  (2025-05-30T03:43:24Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [83.18146814163308]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation [22.875285119636235]
  R1 distillation scheme has emerged as a promising approach for training cost-effective models with enhanced reasoning abilities.<n>This study examines the universality of distillation data and identifies key components that enable the efficient transfer of long-chain reasoning capabilities.<n>We propose DLCoT (Deconstructing Long Chain-of-Thought), a distillation data enhancement framework.
  arXiv  Detail & Related papers  (2025-03-20T17:46:38Z)
• When More is Less: Understanding Chain-of-Thought Length in LLMs [51.631483479081645]
  Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems.<n>This paper argues that longer CoTs are often presumed superior, arguing that longer is not always better.
  arXiv  Detail & Related papers  (2025-02-11T05:28:59Z)
• Unveiling the Mechanisms of Explicit CoT Training: How CoT Enhances Reasoning Generalization [9.191236388401226]
  The integration of explicit Chain-of-Thought (CoT) reasoning into training large language models has advanced their reasoning capabilities, yet the mechanisms by which CoT enhances generalization remain poorly understood.<n>This work investigates (1) textithow CoT training reshapes internal model representations and (2) textitwhy it improves both in-distribution (ID) and out-of-distribution (OOD) reasoning generalization.
  arXiv  Detail & Related papers  (2025-02-07T05:21:13Z)
• A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning [48.51969964676017]
  Chain-of-Thought (CoT) holds a significant place in augmenting the reasoning performance for large language models. We propose a Read-and-Control approach for controlling the accuracy of CoT.
  arXiv  Detail & Related papers  (2024-06-18T04:07:13Z)
• Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness [17.6082037230676]
  Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks.<n>Previous work attempts to evaluate it but falls short in providing an in-depth analysis of patterns that influence the CoT.<n>We identify key factors that influence CoT effectiveness on performance improvement, including problem difficulty, information gain, and information flow.
  arXiv  Detail & Related papers  (2024-05-29T09:17:46Z)
• Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection [73.31406286956535]
  We introduce the Ladder-of-Thought (LoT) for the stance detection task. LoT directs the small LMs to assimilate high-quality external knowledge, refining the intermediate rationales produced. Our empirical evaluations underscore LoT's efficacy, marking a 16% improvement over GPT-3.5 and a 10% enhancement compared to GPT-3.5 with CoT on stance detection task.
  arXiv  Detail & Related papers  (2023-08-31T14:31:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.