Hierarchical Chain-of-Thought Prompting: Enhancing LLM Reasoning Performance and Efficiency

• URL: http://arxiv.org/abs/2604.00130v1
• Date: Tue, 31 Mar 2026 18:35:22 GMT
• Title: Hierarchical Chain-of-Thought Prompting: Enhancing LLM Reasoning Performance and Efficiency
• Abstract summary: Chain-of-Thought (CoT) prompting has significantly improved the reasoning capabilities of large language models (LLMs)<n>In this work, we introduce Hierarchical Chain-of-Thought (Hi-CoT) prompting.<n>Hi-CoT decomposes the reasoning process into hierarchical substeps by alternating between instructional planning and step-by-step execution.
• Score: 4.139013391483979
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-Thought (CoT) prompting has significantly improved the reasoning capabilities of large language models (LLMs). However, conventional CoT often relies on unstructured, flat reasoning chains that suffer from redundancy and suboptimal performance. In this work, we introduce Hierarchical Chain-of-Thought (Hi-CoT) prompting, a structured reasoning paradigm specifically designed to address the challenges of complex, multi-step reasoning. Hi-CoT decomposes the reasoning process into hierarchical substeps by alternating between instructional planning and step-by-step execution. This decomposition enables LLMs to better manage long reasoning horizons and maintain logical coherence. Extensive evaluations across diverse LLMs and mathematical reasoning benchmarks show that Hi-CoT consistently improves average accuracy by 6.2% (up to 61.4% on certain models and tasks) while reducing reasoning trace length by 13.9% compared to CoT prompting. We further show that accuracy and efficiency are maximized when models strictly adhere to the hierarchical structure. Our code is available at https://github.com/XingshuaiHuang/Hi-CoT.

Related papers

• ChainPrune: Evaluating and Reducing Redundancy in Long Chain-of-Thought Reasoning [48.3829293800053]
  We propose ChainPrune, a novel reasoning path semantic structural optimization method.<n>We consolidate self-generated reasoning paths into a tree-based structure, followed by a multi-criteria dominant path selection process.<n>We incorporate a DPO-based preference learning method combined with supervised loss, effectively mitigating false reward suppression.
  arXiv  Detail & Related papers  (2026-08-22T09:09:05Z)
• Cognitive Loop of Thought: Reversible Hierarchical Markov Chain for Efficient Mathematical Reasoning [11.603283873843294]
  Multi-step Chain-of-Thought (CoT) has significantly advanced the mathematical reasoning capabilities of LLMs.<n>Existing approaches attempt to alleviate this by reducing KV Cache redundancy via Markov chain-like structures.<n>We propose a novel Chain-of-Thought framework based on Reversible Hierarchical Markov Chain, Cognitive Loop of Thought (CLoT)
  arXiv  Detail & Related papers  (2026-04-08T08:17:38Z)
• Short Chains, Deep Thoughts: Balancing Reasoning Efficiency and Intra-Segment Capability via Split-Merge Optimization [68.89915707647138]
  Large Reasoning Models (LRMs) have demonstrated impressive capabilities in solving complex tasks through the generation of long reasoning chains.<n>We propose textbfCoSMo (textbfSplit-textbfMerge textbfOptimization), a framework designed to eliminate structural redundancy rather than indiscriminately restricting token volume.
  arXiv  Detail & Related papers  (2026-02-03T05:54:28Z)
• Reinforced Efficient Reasoning via Semantically Diverse Exploration [73.41112984160992]
  Reinforcement learning with verifiable rewards (RLVR) has proven effective in enhancing the reasoning of large language models (LLMs)<n>We propose reinforced efficient reasoning via semantically diverse explorations, i.e., ROSE, for LLMs.<n>Our method incorporates a semantic-entropy-based branching strategy and an $varepsilon$-exploration mechanism.
  arXiv  Detail & Related papers  (2026-01-08T15:56:44Z)
• AutoL2S: Auto Long-Short Reasoning for Efficient Large Language Models [56.063571989395946]
  The reasoning-capable large language models (LLMs) demonstrate strong performance on complex reasoning tasks.<n>Recent approaches attempt to address this challenge by manually deciding when to apply long or short reasoning.<n>We propose Auto Long-Short Reasoning (AutoL2S), a dynamic and model-agnostic framework that enables LLMs to dynamically compress their generated reasoning path.
  arXiv  Detail & Related papers  (2025-05-28T17:59:53Z)
• Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning [15.137717200618454]
  Prune-on-Logic is a framework that transforms Long-CoT into logic graphs and selectively prunes low-utility reasoning steps.<n>We find that verification pruning consistently improves accuracy while reducing token usage, whereas reasoning or indiscriminate pruning degrades performance.
  arXiv  Detail & Related papers  (2025-05-20T16:38:32Z)
• AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning [30.265984245328124]
  Chain-of-Thought prompting indiscriminately generates lengthy reasoning steps for all queries.<n>AdaCoT (Adaptive Chain-of-Thought) is a novel framework enabling LLMs to adaptively decide when to invoke CoT.<n>A key technical contribution is Selective Loss Masking (SLM), designed to counteract decision boundary collapse.
  arXiv  Detail & Related papers  (2025-05-17T08:27:00Z)
• Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization [86.56120216550232]
  We propose a novel two-stage framework for adaptive and efficient reasoning.<n>First, we construct a hybrid reasoning model by merging long and short CoT models.<n>Second, we apply bi-level preference training to guide the model to select suitable reasoning styles.
  arXiv  Detail & Related papers  (2025-04-30T14:01:45Z)
• Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching [64.74765550805024]
  Chain-of-Thought prompting elicits step-by-step problem solving, but often at the cost of excessive verbosity in intermediate outputs.<n>We propose Sketch-of-Thought (SoT), a prompting framework that integrates cognitively inspired reasoning paradigms with linguistic constraints.<n>SoT achieves token reductions of up to 84% with minimal accuracy loss across 18 reasoning datasets.
  arXiv  Detail & Related papers  (2025-03-07T06:57:17Z)
• When More is Less: Understanding Chain-of-Thought Length in LLMs [51.631483479081645]
  Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems.<n>This paper argues that longer CoTs are often presumed superior, arguing that longer is not always better.
  arXiv  Detail & Related papers  (2025-02-11T05:28:59Z)
• Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs [37.147529569445396]
  Tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. Fine-tuning language models (LLMs) leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT.
  arXiv  Detail & Related papers  (2024-06-13T14:07:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.