Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention

• URL: http://arxiv.org/abs/2601.09805v1
• Date: Wed, 14 Jan 2026 19:10:10 GMT
• Title: Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention
• Authors: Nguyen Minh Phuong, Dang Huu Tien, Naoya Inoue,
• Abstract summary: A non-interactive, end-to-end framework enables reasoning to emerge within the model itself.<n>We show that introducing structural information into the few-shot prompt activates a subset of attention heads that patterns aligned with logical reasoning operators.<n>We propose Attention-Aware Intervention (AAI), an inference-time intervention method that reweights attention scores across selected heads identified by their logical patterns.
• Score: 4.584629831500306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern logical reasoning with LLMs primarily relies on employing complex interactive frameworks that decompose the reasoning process into subtasks solved through carefully designed prompts or requiring external resources (e.g., symbolic solvers) to exploit their strong logical structures. While interactive approaches introduce additional overhead, hybrid approaches depend on external components, which limit their scalability. A non-interactive, end-to-end framework enables reasoning to emerge within the model itself -- improving generalization while preserving analyzability without any external resources. In this work, we introduce a non-interactive, end-to-end framework for reasoning tasks. We show that introducing structural information into the few-shot prompt activates a subset of attention heads that patterns aligned with logical reasoning operators. Building on this insight, we propose Attention-Aware Intervention (AAI), an inference-time intervention method that reweights attention scores across selected heads identified by their logical patterns. AAI offers an efficient way to steer the model's reasoning toward leveraging prior knowledge through attention modulation. Extensive experiments show that AAI enhances logical reasoning performance across diverse benchmarks and model architectures, while incurring negligible additional computational overhead. Code is available at https://github.com/phuongnm94/aai_for_logical_reasoning.

Related papers

• ActivationReasoning: Logical Reasoning in Latent Activation Spaces [43.17973499652433]
  Large language models (LLMs) excel at generating fluent text, but their internal reasoning remains opaque and difficult to control.<n>We introduce ActivationReasoning (AR), a framework that embeds explicit logical reasoning into the latent space of LLMs.<n>AR scales robustly with reasoning complexity, generalizes to abstract and context-sensitive tasks, and transfers across model backbones.
  arXiv  Detail & Related papers  (2025-10-21T00:21:04Z)
• Agentic AI Reasoning for Mobile Edge General Intelligence: Fundamentals, Approaches, and Directions [74.35421055079655]
  Large language models (LLMs) have enabled an emergence of agentic artificial intelligence (AI) with powerful reasoning and autonomous decision-making capabilities.<n>Mobile Edge General Intelligence (MEGI) brings real-time, privacy-preserving reasoning to the network edge.<n>We propose a joint optimization framework for efficient LLM reasoning deployment in MEGI.
  arXiv  Detail & Related papers  (2025-09-27T10:53:48Z)
• Implicit Reasoning in Large Language Models: A Comprehensive Survey [67.53966514728383]
  Large Language Models (LLMs) have demonstrated strong generalization across a wide range of tasks.<n>Recent studies have shifted attention from explicit chain-of-thought prompting toward implicit reasoning.<n>This survey introduces a taxonomy centered on execution paradigms, shifting the focus from representational forms to computational strategies.
  arXiv  Detail & Related papers  (2025-09-02T14:16:02Z)
• Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities at solving complex reasoning tasks with Chain-of-Thought prompting.<n>We introduce textbfinverse reasoning, a novel paradigm enabling LLMs to decompose and explain their own reasoning chains post-hoc.<n>Our work creates new avenues for transparent AI systems and closes significant gaps in AI safety, education, and scientific discovery.
  arXiv  Detail & Related papers  (2025-06-30T09:53:41Z)
• KAG-Thinker: Interactive Thinking and Deep Reasoning in LLMs via Knowledge-Augmented Generation [35.555200530999365]
  We introduce KAG-Thinker, which upgrade KAG to a multi-turn interactive thinking and deep reasoning framework powered by a dedicated parameter-light large language model (LLM)<n>Our approach constructs a structured thinking process for solving complex problems, enhancing the the logical coherence and contextual consistency of the reasoning process.
  arXiv  Detail & Related papers  (2025-06-21T14:58:53Z)
• PixelThink: Towards Efficient Chain-of-Pixel Reasoning [70.32510083790069]
  PixelThink is a simple yet effective scheme that integrates externally estimated task difficulty and internally measured model uncertainty.<n>It learns to compress reasoning length in accordance with scene complexity and predictive confidence.<n> Experimental results demonstrate that the proposed approach improves both reasoning efficiency and overall segmentation performance.
  arXiv  Detail & Related papers  (2025-05-29T17:55:49Z)
• Efficient Inference for Large Reasoning Models: A Survey [74.17203483365171]
  Large Reasoning Models (LRMs) significantly improve the reasoning ability of Large Language Models (LLMs) by learning to reason.<n>However, their deliberative reasoning process leads to inefficiencies in token usage, memory consumption, and inference time.<n>This survey provides a review of efficient inference methods designed specifically for LRMs, focusing on mitigating token inefficiency while preserving the reasoning quality.
  arXiv  Detail & Related papers  (2025-03-29T13:27:46Z)
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models [49.61246073215651]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in complex tasks.<n>Recent advancements in OpenAI o1 and DeepSeek-R1 have further improved performance in System-2 reasoning domains.<n>However, they also introduce significant computational overhead due to verbose and redundant outputs.
  arXiv  Detail & Related papers  (2025-03-20T17:59:38Z)
• From Chaos to Order: The Atomic Reasoner Framework for Fine-grained Reasoning in Large Language Models [46.02816479205161]
  We present textbfAtomic Reasoner (textbfAR), a cognitive inference strategy that enables fine-grained reasoning.<n>AR decomposes the reasoning process into atomic cognitive units, employing a cognitive routing mechanism.<n>Results show AR's superior reasoning capabilities without the computational burden of exhaustive solution searches.
  arXiv  Detail & Related papers  (2025-03-20T08:34:53Z)
• Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools [19.70178343422698]
  We introduce Agentic Reasoning, a framework that enhances large language model (LLM) reasoning by integrating external tool-using agents.<n>Key innovation in our framework is the Mind-Map agent, which constructs a structured knowledge graph to store reasoning context.<n>When deployed on DeepSeek-R1, our method achieves a new state-of-the-art (SOTA) among public models.
  arXiv  Detail & Related papers  (2025-02-07T04:08:46Z)
• Logic Agent: Enhancing Validity with Logic Rule Invocation [24.815341366820753]
  Chain-of-Thought prompting has emerged as a pivotal technique for augmenting the inferential capabilities of language models during reasoning tasks.<n>This paper introduces the Logic Agent (LA), an agent-based framework aimed at enhancing the validity of reasoning processes in Large Language Models.
  arXiv  Detail & Related papers  (2024-04-28T10:02:28Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• LogiGAN: Learning Logical Reasoning via Adversarial Pre-training [58.11043285534766]
  We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models. Inspired by the facilitation effect of reflective thinking in human learning, we simulate the learning-thinking process with an adversarial Generator-Verifier architecture. Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets.
  arXiv  Detail & Related papers  (2022-05-18T08:46:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.