C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

• URL: http://arxiv.org/abs/2412.11664v1
• Date: Mon, 16 Dec 2024 11:12:45 GMT
• Title: C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
• Authors: Yu Kang, Xianghui Sun, Liangyu Chen, Wei Zou,
• Abstract summary: Chain-of-Thought (CoT) before deriving the answer can improve the reasoning capabilities of large language models (LLMs)<n>However, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs.<n>This paper presents a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT.
• Score: 18.073777359647515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating Chain-of-Thought (CoT) before deriving the answer can effectively improve the reasoning capabilities of large language models (LLMs) and significantly improve the accuracy of the generated answer. However, in most cases, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs. Furthermore, existing research has discovered that shortening the reasoning steps in CoT, even while preserving the key information, diminishes LLMs' abilities. These phenomena make it difficult to use LLMs and CoT in many real-world applications that only require the final answer and are sensitive to latency, such as search and recommendation. To reduce the costs of model decoding and shorten the length of the generated CoT, this paper presents $\textbf{C}$onditioned $\textbf{C}$ompressed $\textbf{C}$hain-of-$\textbf{T}$hought (C3oT), a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT while maintaining key information and interpretability, a conditioned training method to train LLMs with both longer CoT and shorter CoT simultaneously to learn the corresponding relationships between them, and a conditioned inference method to gain the reasoning ability learned from longer CoT by generating shorter CoT. We conduct experiments over four datasets from arithmetic and commonsense scenarios, showing that the proposed method is capable of compressing the length of generated CoT by up to more than 50% without compromising its effectiveness.

Related papers

• Hawkeye:Efficient Reasoning with Model Collaboration [7.26791045376255]
  Chain-of-Thought (CoT) reasoning has demonstrated remarkable effectiveness in enhancing the reasoning abilities of large language models (LLMs) Most CoT tokens are unnecessary, and retaining only a small portion of them is sufficient for producing high-quality responses. We propose HAWKEYE, a novel post-training and inference framework where a large model produces concise CoT instructions to guide a smaller model in response generation.
  arXiv  Detail & Related papers  (2025-04-01T05:09:04Z)
• Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering [59.34894142132706]
  Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. We propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-14T11:30:37Z)
• TokenSkip: Controllable Chain-of-Thought Compression in LLMs [11.583847083770031]
  Chain-of-Thought (CoT) has been proven effective in enhancing the reasoning capabilities of large language models (LLMs) We propose TokenSkip, a simple yet effective approach that enables LLMs to selectively skip less important tokens, allowing for controllable CoT compression.
  arXiv  Detail & Related papers  (2025-02-17T17:37:26Z)
• CoT-Valve: Length-Compressible Chain-of-Thought Tuning [50.196317781229496]
  We introduce a new tuning and inference strategy named CoT-Valve, designed to allow models to generate reasoning chains of varying lengths. We show that CoT-Valve successfully enables controllability and compressibility of the chain and shows better performance than the prompt-based control.
  arXiv  Detail & Related papers  (2025-02-13T18:52:36Z)
• When More is Less: Understanding Chain-of-Thought Length in LLMs [53.77747102201451]
  Chain-of-thought (CoT) reasoning enhances the multi-step reasoning capabilities of large language models (LLMs) However, for most models and tasks, does an increase in CoT length consistently lead to improved reasoning accuracy? In this paper, we observe a nuanced relationship: as the number of reasoning steps increases, performance initially improves but eventually decreases.
  arXiv  Detail & Related papers  (2025-02-11T05:28:59Z)
• Efficient Long Context Language Model Retrieval with Compression [57.09163579304332]
  Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR) We propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages. We show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91.
  arXiv  Detail & Related papers  (2024-12-24T07:30:55Z)
• From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency [17.612497960364916]
  Chain-of-thought (CoT) significantly enhances the reasoning performance of large language models (LLM) We demonstrate that CoT can substantially improve sample efficiency even when representation power is sufficient. We show that CoT simplifies the learning process by introducing sparse dependencies among input tokens, and leads to a sparse and interpretable attention.
  arXiv  Detail & Related papers  (2024-10-07T19:45:09Z)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning [55.52872152909785]
  Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs) We show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks.
  arXiv  Detail & Related papers  (2024-09-18T17:55:00Z)
• Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding [14.175444025026508]
  Large language models (LLMs) have demonstrated remarkable capabilities in tasks requiring chain-of-thought (CoT) prompting. generating the full CoT process results in significantly longer output sequences, leading to increased computational costs and latency during inference. We propose a novel approach to compress the CoT process through semantic alignment, enabling more efficient decoding while preserving the benefits of CoT reasoning.
  arXiv  Detail & Related papers  (2024-09-13T06:29:20Z)
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting [124.69672273754144]
  Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs) Existing CoT approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. We introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts.
  arXiv  Detail & Related papers  (2024-03-21T11:34:26Z)
• Chain-of-Thought Reasoning Without Prompting [40.92854235219315]
  CoT reasoning paths can be elicited from pre-trained language models by simply altering the textitdecoding process. The presence of a CoT in the decoding path correlates with a higher confidence in the model's decoded answer.
  arXiv  Detail & Related papers  (2024-02-15T18:55:41Z)
• Stress Testing Chain-of-Thought Prompting for Large Language Models [0.16317061277456998]
  This report examines the effectiveness of Chain-of-Thought (CoT) prompting in improving the multi-step reasoning abilities of large language models (LLMs) We analyze the impact of three types of CoT prompt perturbations, namely CoT order, CoT values, and CoT operators on the performance of GPT-3 on various tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:21:33Z)
• T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering [59.63860993280275]
  Large Language Models (LLMs) have demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. We propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. Our approach achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%.
  arXiv  Detail & Related papers  (2023-05-05T11:56:30Z)
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions [50.114651561111245]
  We propose IRCoT, a new approach for multi-step question answering. It interleaves retrieval with steps in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT.
  arXiv  Detail & Related papers  (2022-12-20T18:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.