Calibrating Reasoning in Language Models with Internal Consistency

• URL: http://arxiv.org/abs/2405.18711v2
• Date: Thu, 05 Dec 2024 04:01:28 GMT
• Title: Calibrating Reasoning in Language Models with Internal Consistency
• Authors: Zhihui Xie, Jizhou Guo, Tong Yu, Shuai Li,
• Abstract summary: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks.<n>LLMs often generate text with obvious mistakes and contradictions.<n>In this work, we investigate reasoning in LLMs through the lens of internal representations.
• Score: 18.24350001344488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks, aided by techniques like chain-of-thought prompting that elicits verbalized reasoning. However, LLMs often generate text with obvious mistakes and contradictions, raising doubts about their ability to robustly process and utilize generated rationales. In this work, we investigate reasoning in LLMs through the lens of internal representations, focusing on how these representations are influenced by generated rationales. Our preliminary analysis reveals that while generated rationales improve answer accuracy, inconsistencies emerge between the model's internal representations in middle layers and those in final layers, potentially undermining the reliability of their reasoning processes. To address this, we propose internal consistency as a measure of the model's confidence by examining the agreement of latent predictions decoded from intermediate layers. Extensive empirical studies across different models and datasets demonstrate that internal consistency effectively distinguishes between correct and incorrect reasoning paths. Motivated by this, we propose a new approach to calibrate reasoning by up-weighting reasoning paths with high internal consistency, resulting in a significant boost in reasoning performance. Further analysis uncovers distinct patterns in attention and feed-forward modules across layers, providing insights into the emergence of internal inconsistency. In summary, our results demonstrate the potential of using internal representations for self-evaluation of LLMs. Our code is available at github.com/zhxieml/internal-consistency.

Related papers

• CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching [50.65932158912512]
  We propose a new causal reasoning benchmark, CausalFlip, to encourage the development of new large language models.<n>CaulFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations.<n>We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought supervision, and a proposed internalized causal reasoning approach.
  arXiv  Detail & Related papers  (2026-02-23T18:06:15Z)
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models [77.98801218316505]
  Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning.<n>We investigate the internal processing of LLMs during in-context concept inference.
  arXiv  Detail & Related papers  (2026-02-08T03:14:39Z)
• Interpreting and Controlling LLM Reasoning through Integrated Policy Gradient [27.26870804635122]
  Large language models (LLMs) demonstrate strong reasoning abilities in solving complex real-world problems.<n> internal mechanisms driving these complex reasoning behaviors remain opaque.<n>We propose Integrated Policy Gradient (IPG), a novel framework that attributes reasoning behaviors to model's inner components.
  arXiv  Detail & Related papers  (2026-02-02T16:43:09Z)
• Reasoning as State Transition: A Representational Analysis of Reasoning Evolution in Large Language Models [50.39102836928242]
  We introduce a representational perspective to investigate the dynamics of the model's internal states.<n>We discover that post-training yields only limited improvement in static initial representation quality.
  arXiv  Detail & Related papers  (2026-01-31T15:23:33Z)
• How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns [51.02752099869218]
  Large Language Models (LLMs) display strikingly different generalization behaviors.<n>We introduce a novel benchmark that decomposes reasoning into atomic core skills.<n>We show that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns.
  arXiv  Detail & Related papers  (2025-12-30T08:16:20Z)
• Implicit Reasoning in Large Language Models: A Comprehensive Survey [67.53966514728383]
  Large Language Models (LLMs) have demonstrated strong generalization across a wide range of tasks.<n>Recent studies have shifted attention from explicit chain-of-thought prompting toward implicit reasoning.<n>This survey introduces a taxonomy centered on execution paradigms, shifting the focus from representational forms to computational strategies.
  arXiv  Detail & Related papers  (2025-09-02T14:16:02Z)
• Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning [33.30315111732609]
  Chain of Thought (CoT) reasoning has demonstrated remarkable deep reasoning capabilities.<n>However, its reliability is often undermined by the accumulation of errors in intermediate steps.<n>This paper introduces an approach to calibrate the CoT reasoning accuracy by leveraging the model's intrinsic veracity encoding.
  arXiv  Detail & Related papers  (2025-07-14T07:41:35Z)
• Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability [70.4107059502882]
  Training language models with rationales augmentation has been shown to be beneficial in many existing works.<n>We conduct comprehensive investigations to thoroughly inspect the impact of rationales on model performance.
  arXiv  Detail & Related papers  (2025-05-30T02:39:37Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [53.18562650350898]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling [56.26834106704781]
  Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs)<n>We provide evidence supporting the presence of LLMs' internal compass that dictate the correctness of factual recall at the time of generation.<n>Scaling experiments across model sizes and training dynamics highlight that self-awareness emerges rapidly during training and peaks in intermediate layers.
  arXiv  Detail & Related papers  (2025-05-27T16:24:02Z)
• Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM [11.181783720439563]
  Large Language Models (LLMs) display sophisticated reasoning abilities via extended Chain-of-Thought (CoT) generation.<n>RLMs often demonstrate counterintuitive and unstable behaviors, such as performance degradation under few-shot prompting.<n>We introduce a unified graph-based analytical framework for better modeling the reasoning processes of RLMs.
  arXiv  Detail & Related papers  (2025-05-20T03:54:57Z)
• A Modular Dataset to Demonstrate LLM Abstraction Capability [3.0899016152680754]
  Large language models (LLMs) exhibit impressive capabilities but struggle with reasoning errors due to hallucinations and flawed logic. We introduce ArrangementPuzzle, a novel puzzle dataset with structured solutions and automated stepwise correctness verification. We trained a classifier model on LLM activations on this dataset and found that it achieved over 80% accuracy in predicting reasoning correctness.
  arXiv  Detail & Related papers  (2025-03-22T04:25:30Z)
• Demystifying the Accuracy-Interpretability Trade-Off: A Case Study of Inferring Ratings from Reviews [11.905166272929263]
  Interpretable machine learning models offer understandable reasoning behind their decision-making process. This trade-off between interpretability and model performance has sparked discussions around the deployment of AI.
  arXiv  Detail & Related papers  (2025-03-10T23:17:46Z)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument [71.33552795870544]
  Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks. They can be easily misled by unfaithful arguments during conversations, even when their original statements are correct. We propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation.
  arXiv  Detail & Related papers  (2025-01-02T16:38:21Z)
• Internal Consistency and Self-Feedback in Large Language Models: A Survey [19.647988281648253]
  We use a unified perspective of internal consistency, offering explanations for reasoning deficiencies and hallucinations. We introduce an effective theoretical framework capable of mining internal consistency, named Self-Feedback.
  arXiv  Detail & Related papers  (2024-07-19T17:59:03Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Evaluating Consistency and Reasoning Capabilities of Large Language Models [0.0]
  Large Language Models (LLMs) are extensively used today across various sectors, including academia, research, business, and finance. Despite their widespread adoption, these models often produce incorrect and misleading information, exhibiting a tendency to hallucinate. This paper aims to evaluate and compare the consistency and reasoning capabilities of both public and proprietary LLMs.
  arXiv  Detail & Related papers  (2024-04-25T10:03:14Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) can estimate causal effects under interventions on different parts of a system. We conduct empirical analyses to evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention. We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types, and enable a study of intervention-based reasoning.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning [25.732397636695882]
  We show that large language models (LLMs) display reasoning patterns akin to those observed in humans. Our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning.
  arXiv  Detail & Related papers  (2024-02-20T12:58:14Z)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention [53.896974148579346]
  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. The enigmatic black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. We propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs.
  arXiv  Detail & Related papers  (2023-12-22T19:55:58Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)
• From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning [66.98861219674039]
  Heuristic-Analytic Reasoning (HAR) strategies drastically improve the coherence of rationalizations for model decisions. Our findings suggest that human-like reasoning strategies can effectively improve the coherence and reliability of PLM reasoning.
  arXiv  Detail & Related papers  (2023-10-24T19:46:04Z)
• Question Decomposition Improves the Faithfulness of Model-Generated Reasoning [23.34325378824462]
  Large language models (LLMs) are difficult to verify the correctness and safety of their behavior. One approach is to prompt LLMs to externalize their reasoning, by having them generate step-by-step reasoning as they answer a question. This approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT.
  arXiv  Detail & Related papers  (2023-07-17T00:54:10Z)
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing [139.77117915309023]
  CRITIC allows large language models to validate and amend their own outputs in a manner similar to human interaction with tools. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs.
  arXiv  Detail & Related papers  (2023-05-19T15:19:44Z)
• Competence-Based Analysis of Language Models [21.43498764977656]
  CALM (Competence-based Analysis of Language Models) is designed to investigate LLM competence in the context of specific tasks. We develop a new approach for performing causal probing interventions using gradient-based adversarial attacks. We carry out a case study of CALM using these interventions to analyze and compare LLM competence across a variety of lexical inference tasks.
  arXiv  Detail & Related papers  (2023-03-01T08:53:36Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.