Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

• URL: http://arxiv.org/abs/2512.21002v1
• Date: Wed, 24 Dec 2025 06:57:35 GMT
• Title: Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
• Authors: Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi, Hejian Sang, Shao Tang, Qingquan Song, Zhipeng Wang, Muhammad Abdul-Mageed,
• Abstract summary: Distilling the reasoning capabilities from a large language model to a smaller student model often involves training on substantial amounts of reasoning data.<n>We investigate how the allocation of supervision across different segments (P, CoT, A) affects student performance.<n>We observe that training on only the first $50%$ of tokens of every training sequence can retain, on average, $approx94%$ of full-sequence performance.
• Score: 27.167625134416156
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Distilling the reasoning capabilities from a large language model (LLM) to a smaller student model often involves training on substantial amounts of reasoning data. However, distillation over lengthy sequences with prompt (P), chain-of-thought (CoT), and answer (A) segments makes the process computationally expensive. In this work, we investigate how the allocation of supervision across different segments (P, CoT, A) affects student performance. Our analysis shows that selective knowledge distillation over only the CoT tokens can be effective when the prompt and answer information is encompassed by it. Building on this insight, we establish a truncation protocol to quantify computation-quality tradeoffs as a function of sequence length. We observe that training on only the first $50\%$ of tokens of every training sequence can retain, on average, $\approx94\%$ of full-sequence performance on math benchmarks while reducing training time, memory usage, and FLOPs by about $50\%$ each. These findings suggest that reasoning distillation benefits from prioritizing early reasoning tokens and provides a simple lever for computation-quality tradeoffs. Codes are available at https://github.com/weiruichen01/distilling-the-essence.

Related papers

• Decomposing Reasoning Efficiency in Large Language Models [2.4149105714758545]
  We decompose token efficiency into interpretable factors: completion under a fixed token budget, conditional correctness given completion, and verbosity.<n>When reasoning traces are available, we add deterministic trace-quality measures to separate looping from verbose-but-engaged reasoning.<n>Our decomposition reveals distinct bottleneck profiles that suggest different efficiency interventions.
  arXiv  Detail & Related papers  (2026-02-10T14:09:18Z)
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs [57.217593337454026]
  TokenSqueeze is a novel Long2Short method that condenses reasoning paths while preserving performance and relying exclusively on self-generated data.<n>We show that TokenSqueeze reduces token usage while maintaining accuracy on the MATH500 benchmark.
  arXiv  Detail & Related papers  (2025-11-17T10:38:56Z)
• Rethinking Thinking Tokens: LLMs as Improvement Operators [80.12087211785949]
  Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which allows them to explore solution strategies with self-checking.<n>This results in higher accuracy, but inflates context length, token/compute cost, and answer latency.<n>We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier?<n>We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace
  arXiv  Detail & Related papers  (2025-10-01T17:08:59Z)
• Train Long, Think Short: Curriculum Learning for Efficient Reasoning [51.506559652495476]
  We propose a curriculum learning strategy for length-controlled reasoning.<n>Our method starts with generous token budgets and gradually tightens them over training.<n>Experiments on GSM8K, MATH500, SVAMP, College Math, and GSM+ demonstrate that curriculum-based training consistently outperforms fixed-budget baselines.
  arXiv  Detail & Related papers  (2025-08-12T13:48:03Z)
• Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion [34.582439587552656]
  We investigate whether the success of a zero-shot Chain-of-Thought process can be predicted before completion.<n>We find that a probing classifier, based on LLM representations, performs well even before a single token is generated.
  arXiv  Detail & Related papers  (2025-05-30T08:54:28Z)
• Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster [51.89995713333108]
  Chain-of-thought (CoT) distillation allows a large language model (LLM) to guide a small language model (SLM) in reasoning tasks.<n>Existing methods train the SLM to learn the long rationale in one iteration.<n>We propose chunk-wise training (CWT), which uses a search to divide the rationale into internal semantically coherent chunks.
  arXiv  Detail & Related papers  (2025-05-24T11:04:52Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Investigating Mysteries of CoT-Augmented Distillation [24.33660998599006]
  Eliciting "chain of thought" (CoT) rationales has been shown to consistently improve LLM performance on tasks like question answering. We ask: Why and how does this additional training signal help in model distillation?
  arXiv  Detail & Related papers  (2024-06-20T17:15:46Z)
• SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference [17.947904697850433]
  We present SkipDecode, a token-level early exit method for batch inferencing and KeyValue caching. It overcomes prior constraints by setting up singular-level exit point for every token in a batch at each sequence position. It also guarantees a monotonic decrease in exit points, thereby eliminating the need to recompute KV Caches for preceding tokens.
  arXiv  Detail & Related papers  (2023-07-05T19:59:09Z)
• AdapLeR: Speeding up Inference by Adaptive Length Reduction [15.57872065467772]
  We propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance. Our method dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. Our experiments on several diverse classification tasks show speedups up to 22x during inference time without much sacrifice in performance.
  arXiv  Detail & Related papers  (2022-03-16T23:41:38Z)
• You Only Need End-to-End Training for Long-Tailed Recognition [8.789819609485225]
  Cross-entropy loss tends to produce highly correlated features on imbalanced data. We propose two novel modules, Block-based Relatively Balanced Batch Sampler (B3RS) and Batch Embedded Training (BET) Experimental results on the long-tailed classification benchmarks, CIFAR-LT and ImageNet-LT, demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2021-12-11T11:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.