Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers

• URL: http://arxiv.org/abs/2606.21678v1
• Date: Fri, 19 Jun 2026 18:37:24 GMT
• Title: Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers
• Abstract summary: Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning.<n>We propose verifier-coupled reasoning, a framework that inserts inline claims into reasoning traces and trains an auxiliary consistency head to predict programmatic verifier outputs.<n>We find a gap between decodability and faithfulness: consistency training reliably makes verifier information decodable from rationale representations, but decodability does not guarantee faithful generation.
• Score: 0.7212939068975618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning. We propose verifier-coupled reasoning, a framework that inserts inline claims into reasoning traces and trains an auxiliary consistency head to predict programmatic verifier outputs from rationale-span hidden states. The central finding is a gap between decodability and faithfulness: consistency training reliably makes verifier information decodable from rationale representations, but decodability does not guarantee faithful generation. In LeanCheck (formal theorem proving), rationale-only and proof-only pooling achieve perfect directional separation under counterfactual conflict. In KataGo (Go engine), commentary spans encode 10-way win-rate buckets at 81% accuracy. Yet in a code setting, the model achieves 98.6% coupling while its generated explanations remain unfaithful: fluent prose with correct structured claims, but describing unrelated algorithms; a controlled pretrained-vs-from-scratch comparison shows the gap is not capacity-driven. Synthetic activation patching confirms causal influence (73-89% vs. 31% baseline), FEVER reveals that evidence-only pooling isolates genuine evidence sensitivity at the cost of raw accuracy, and per-claim analysis shows that consistency loss disproportionately benefits fine-grained claims over binary ones. These results establish that consistency losses are effective diagnostics and representation-shaping tools, but not sufficient conditions for faithful reasoning.

Related papers

• When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models [2.5819252531158683]
  We study logical verification in five open-weight transformer models.<n>Despite near-chance behavioral performance, logical validity is often almost perfectly decodable from hidden states.<n>Our results suggest representing validity, expressing it in behavior, and using it causally are distinct.
  arXiv  Detail & Related papers  (2026-09-02T11:01:09Z)
• The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics [0.9668407688201359]
  Chain-of-thought (CoT) reasoning improves large language model (LLM) performance.<n>Existing approaches that leverage verbalized CoTs to monitor reasoning correctness largely evaluate the semantic correctness or consistency of individual intermediate steps.<n>We show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning.
  arXiv  Detail & Related papers  (2026-08-04T08:04:36Z)
• Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization [10.775710068605006]
  We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem.<n>A full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap.
  arXiv  Detail & Related papers  (2026-06-30T00:27:53Z)
• Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation [0.0]
  We show that models possess the capability to detect fabricated statistics but do not recruit this capability during multi-source synthesis.<n>Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity.
  arXiv  Detail & Related papers  (2026-06-03T20:15:48Z)
• Confidence-Aware Alignment Makes Reasoning LLMs More Reliable [65.44962502963378]
  We introduce CASPO, a framework that aligns token-level confidence with step-wise logical correctness without training a separate reward model.<n>During inference, we propose Confidence-aware Thought (CaT), which dynamically prunes uncertain reasoning branches with negligible O(V) latency.<n>Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency.
  arXiv  Detail & Related papers  (2026-05-08T07:08:25Z)
• Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning [20.336209492846752]
  Formal verification guarantees proof validity but not formalization faithfulness.<n>We investigate whether frontier models exploit this gap when generating Lean 4 proofs.<n>We find no evidence of systematic gaming in unified generation.
  arXiv  Detail & Related papers  (2026-04-21T13:37:49Z)
• FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning [49.65751420291115]
  Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful intermediate steps.<n>We propose FACT-E, a causality-inspired framework for evaluating CoT quality.<n>We show that FACT-E improves reasoning-trajectory selection and yields stronger in-context learnings.
  arXiv  Detail & Related papers  (2026-04-12T15:35:08Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models [5.733004743054914]
  Large Reasoning Models (LRMs) exhibit strong performance, yet often produce rationales that sound plausible but fail to reflect their true decision process.<n>We introduce a formal framework for reasoning faithfulness, defined by two testable conditions.<n>We present RFEval, a benchmark of 7,186 instances that probes faithfulness via controlled, output-level counterfactual interventions.
  arXiv  Detail & Related papers  (2026-02-19T03:49:37Z)
• Latent Veracity Inference for Identifying Errors in Stepwise Reasoning [78.29317733206643]
  We introduce Veracity Search (VS), a discrete search algorithm over veracity assignments.<n>It performs otherwise intractable inference in the posterior distribution over latent veracity values.<n>It generalizes VS, enabling accurate zero-shot veracity inference in novel contexts.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.