Efficient Reasoning with Hidden Thinking

• URL: http://arxiv.org/abs/2501.19201v1
• Date: Fri, 31 Jan 2025 15:10:29 GMT
• Title: Efficient Reasoning with Hidden Thinking
• Authors: Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, Jiuxiang Gu,
• Abstract summary: Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities.<n>We propose $textbfHeima$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space.<n>Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy.
• Score: 48.96945580741641
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose $\textbf{Heima}$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space. We design the Heima Encoder to condense each intermediate CoT into a compact, higher-level hidden representation using a single thinking token, effectively minimizing verbosity and reducing the overall number of tokens required during the reasoning process. Meanwhile, we design corresponding Heima Decoder with traditional Large Language Models (LLMs) to adaptively interpret the hidden representations into variable-length textual sequence, reconstructing reasoning processes that closely resemble the original CoTs. Experimental results across diverse reasoning MLLM benchmarks demonstrate that Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy. Moreover, the effective reconstruction of multimodal reasoning processes with Heima Decoder validates both the robustness and interpretability of our approach.

Related papers

• Efficient Inference for Large Reasoning Models: A Survey [42.61170621552432]
  Large Reasoning Models (LRMs) significantly improve the reasoning ability of Large Language Models (LLMs) by learning to reason. However, their deliberative reasoning process leads to inefficiencies in token usage, memory consumption, and inference time. This survey provides a review of efficient inference methods designed specifically for LRMs, focusing on mitigating token inefficiency while preserving the reasoning quality.
  arXiv  Detail & Related papers  (2025-03-29T13:27:46Z)
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models [54.04678363287392]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in complex tasks. Recent advancements in OpenAI o1 and DeepSeek-R1 have further improved performance in System-2 reasoning domains.
  arXiv  Detail & Related papers  (2025-03-20T17:59:38Z)
• Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering [59.34894142132706]
  Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. We propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-14T11:30:37Z)
• SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs [48.28847964704554]
  Chain-of-Thought (CoT) reasoning enables Large Language Models (LLMs) to solve complex reasoning tasks. We propose a novel approach for continuous-space reasoning that does not require modifying the underlying LLM.
  arXiv  Detail & Related papers  (2025-02-17T18:52:29Z)
• Progressive Multimodal Reasoning via Active Retrieval [64.74746997923967]
  Multi-step multimodal reasoning tasks pose significant challenges for large language models (MLLMs)<n>We propose AR-MCTS, a universal framework designed to progressively improve the reasoning capabilities of MLLMs.<n>We show that AR-MCTS can optimize sampling diversity and accuracy, yielding reliable multimodal reasoning.
  arXiv  Detail & Related papers  (2024-12-19T13:25:39Z)
• Markov Chain of Thought for Efficient Mathematical Reasoning [10.678633785012691]
  Chain of Thought (CoT) of multi-step benefits from the logical structure of the reasoning steps and task-specific actions. We conceptualize the standard multi-step CoT as a novel Markov Chain of Thought (MCoT)
  arXiv  Detail & Related papers  (2024-10-23T07:53:29Z)
• Proof of Thought : Neurosymbolic Program Synthesis allows Robust and Interpretable Reasoning [1.3003982724617653]
  Large Language Models (LLMs) have revolutionized natural language processing, yet they struggle with inconsistent reasoning. This research introduces Proof of Thought, a framework that enhances the reliability and transparency of LLM outputs. Key contributions include a robust type system with sort management for enhanced logical integrity, explicit representation of rules for clear distinction between factual and inferential knowledge.
  arXiv  Detail & Related papers  (2024-09-25T18:35:45Z)
• Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding [14.175444025026508]
  Large language models (LLMs) have demonstrated remarkable capabilities in tasks requiring chain-of-thought (CoT) prompting. generating the full CoT process results in significantly longer output sequences, leading to increased computational costs and latency during inference. We propose a novel approach to compress the CoT process through semantic alignment, enabling more efficient decoding while preserving the benefits of CoT reasoning.
  arXiv  Detail & Related papers  (2024-09-13T06:29:20Z)
• Break the Chain: Large Language Models Can be Shortcut Reasoners [18.047917626825548]
  Chain-of-Thought (CoT) reasoning utilize complex modules but are hampered by high token consumption, limited applicability, and challenges in thinking. This paper conducts a critical evaluation of CoT prompting, extending beyond arithmetic to include complex logical and commonsense reasoning tasks. We propose the integration of human-likes and shortcuts into language models (LMs) through "break the chain" strategies.
  arXiv  Detail & Related papers  (2024-06-04T14:02:53Z)
• Cantor: Inspiring Multimodal Chain-of-Thought of MLLM [83.6663322930814]
  We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. We propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Our experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-04-24T17:59:48Z)
• Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales [102.54274021830207]
  We introduce Fact, a novel paradigm designed to generate multimodal rationales that are faithful, concise, and transferable for teaching MLLMs. We filter rationales that can be transferred to end-to-end paradigms from programming paradigms to guarantee transferability. Our approach also reduces hallucinations owing to its high correlation between images and text.
  arXiv  Detail & Related papers  (2024-04-17T07:20:56Z)
• Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning [74.90592233107712]
  We propose a Direct-Indirect Reasoning (DIR) method, which considers Direct Reasoning (DR) and Indirect Reasoning (IR) as multiple parallel reasoning paths that are merged to derive the final answer.<n>Our DIR method is simple yet effective and can be straightforwardly integrated with existing variants of CoT methods.
  arXiv  Detail & Related papers  (2024-02-06T03:41:12Z)
• Multimodal Chain-of-Thought Reasoning in Language Models [94.70184390935661]
  We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2023-02-02T07:51:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.