From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step

• URL: http://arxiv.org/abs/2405.14838v1
• Date: Thu, 23 May 2024 17:54:14 GMT
• Title: From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step
• Authors: Yuntian Deng, Yejin Choi, Stuart Shieber,
• Abstract summary: In this paper, we investigate if models can be taught to internalize explicit chain-of-thought (CoT) steps. We propose a simple yet effective method for internalizing CoT steps, starting with a model trained for explicit CoT reasoning. Our method proves effective on larger language models, such as Mistral 7B, achieving over 50% accuracy on GSM8K without producing any intermediate steps.
• Score: 47.608403357284026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When leveraging language models for reasoning tasks, generating explicit chain-of-thought (CoT) steps often proves essential for achieving high accuracy in final outputs. In this paper, we investigate if models can be taught to internalize these CoT steps. To this end, we propose a simple yet effective method for internalizing CoT steps: starting with a model trained for explicit CoT reasoning, we gradually remove the intermediate steps and finetune the model. This process allows the model to internalize the intermediate reasoning steps, thus simplifying the reasoning process while maintaining high performance. Our approach enables a GPT-2 Small model to solve 9-by-9 multiplication with up to 99% accuracy, whereas standard training cannot solve beyond 4-by-4 multiplication. Furthermore, our method proves effective on larger language models, such as Mistral 7B, achieving over 50% accuracy on GSM8K without producing any intermediate steps.

Related papers

• StepFun-Prover Preview: Let's Think and Verify Step by Step [14.896796588073725]
  We present StepFun-Prover Preview, a large language model designed for formal theorem proving through tool-integrated reasoning.<n>Our approach enables the model to emulate human-like problem-solving strategies by iteratively refining proofs based on real-time environment feedback.<n>On the miniF2F-test benchmark, StepFun-Prover achieves a pass@1 success rate of $70.0%$.
  arXiv  Detail & Related papers  (2025-07-27T09:38:32Z)
• Dynamic Early Exit in Reasoning Models [13.982812528756504]
  Overthinking in long chain-of-thought (CoT) generation slows down the efficiency of problem solving, but also risks accuracy loss.<n>We propose a simple yet effective method that allows LLMs to self-truncate CoT sequences by early exit during generation.<n>Our method requires no additional training and can be seamlessly integrated into existing o1-like reasoning LLMs.
  arXiv  Detail & Related papers  (2025-04-22T13:36:53Z)
• Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models [56.37421741507468]
  Chain-of-Thought (CoT) reasoning has significantly enhanced the performance of large language models (LLMs) We propose a method to identify critical reasoning steps using perplexity as a measure of their importance.
  arXiv  Detail & Related papers  (2025-02-18T20:04:51Z)
• Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback [94.25162866972077]
  Step-KTO is a training framework that combines process-level and outcome-level binary feedback. Our experiments show that Step-KTO significantly improves both final answer accuracy and the quality of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-01-18T15:38:03Z)
• Truncated Consistency Models [57.50243901368328]
  Training consistency models requires learning to map all intermediate points along PF ODE trajectories to their corresponding endpoints. We empirically find that this training paradigm limits the one-step generation performance of consistency models. We propose a new parameterization of the consistency function and a two-stage training procedure that prevents the truncated-time training from collapsing to a trivial solution.
  arXiv  Detail & Related papers  (2024-10-18T22:38:08Z)
• Fine-Tuning with Divergent Chains of Thought Boosts Reasoning Through Self-Correction in Language Models [63.36637269634553]
  We present a novel method of further improving performance by requiring models to compare multiple reasoning chains. We find that instruction tuning on DCoT datasets boosts the performance of even smaller, and therefore more accessible, language models.
  arXiv  Detail & Related papers  (2024-07-03T15:01:18Z)
• Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation [24.272384832200522]
  We propose mistaktextbfE-textbfDriven key reasontextbfIng step distillatextbfTion (textbfEDIT) We design prompts to generate dual CoTs data with similar reasoning paths but divergent conclusions. Experiments validate the effectiveness of EDIT across both in-domain and out-of-domain benchmark reasoning datasets.
  arXiv  Detail & Related papers  (2024-05-30T06:32:11Z)
• How Do Humans Write Code? Large Models Do It the Same Way Too [14.954886191356342]
  Program-of-Thought (PoT) replaces natural language-based Chain-of-Thought (CoT) as the most popular method in Large Language Models. Using PoT introduces more reasoning errors, such as incorrect formulas or flawed logic, compared to CoT. We propose Human-Think Language (HTL), which leverages a suite of strategies that help integrate PoT and CoT.
  arXiv  Detail & Related papers  (2024-02-24T05:40:01Z)
• Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes [55.2326738851157]
  We introduce Enhancing NumeriCal reasOning with Reliable procEsses (Encore), which derives the reliable reasoning process by decomposing the answer formula. We present a series of pre-training tasks to help models learn the reasoning process generation with synthesized data. Experiments show that Encore yields improvement on all five experimental datasets with an average of 1.8%.
  arXiv  Detail & Related papers  (2024-02-16T13:02:11Z)
• Training Chain-of-Thought via Latent-Variable Inference [30.21067593018967]
  Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a chain-of-thought'' prompt. Naively combining CoT with supervised tuning requires supervision not just of the correct answers, but also of detailed rationales that lead to those answers. We propose a fine-tuning strategy that tries to maximize the emphmarginal log-likelihood of generating a correct answer using CoT prompting.
  arXiv  Detail & Related papers  (2023-11-28T17:47:32Z)
• Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning [75.74103236299477]
  Chain-of-thought prompting(CoT) and tool augmentation have been validated as effective practices for improving large language models. We propose a new approach that can deliberate the reasoning steps with tool interfaces, namely textbfDELI. Experimental results on CARP and six other datasets show that the proposed DELI mostly outperforms competitive baselines.
  arXiv  Detail & Related papers  (2023-06-04T17:02:59Z)
• Distilling Reasoning Capabilities into Smaller Language Models [83.66051257039763]
  Step-by-step reasoning approaches like chain of thought (CoT) have proved to be very effective in inducing reasoning capabilities in large language models. However, the success of the CoT approach is fundamentally tied to the model size, and billion parameter-scale models are often needed to get CoT to work. We propose a knowledge distillation approach that leverages the step-by-step CoT reasoning capabilities of larger models and distills these abilities into smaller models.
  arXiv  Detail & Related papers  (2022-12-01T00:39:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.