Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores

• URL: http://arxiv.org/abs/2608.31068v1
• Date: Mon, 31 Aug 2026 16:43:55 GMT
• Title: Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
• Abstract summary: We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
• Score: 31.709549159768727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.

Related papers

• RePair: Turning Retrieval Failures into Counterfactual Hard Pairs [15.148342372814662]
  RePair mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective.<n>On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples.
  arXiv  Detail & Related papers  (2026-08-30T06:47:38Z)
• The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics [0.9668407688201359]
  Chain-of-thought (CoT) reasoning improves large language model (LLM) performance.<n>Existing approaches that leverage verbalized CoTs to monitor reasoning correctness largely evaluate the semantic correctness or consistency of individual intermediate steps.<n>We show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning.
  arXiv  Detail & Related papers  (2026-08-04T08:04:36Z)
• LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition [55.572260012037084]
  We introduce LC-ERD (Logic-Consistent Endogenous Reward Decomposition), a framework framing self-alignment as latent structure mining.<n>We derive a Variational Logic Potential by aggregating consensus from the model's Latent Logic Expertise.<n>Experiments show LC-ERD delivers a robust self-evolution path, uncovering trade-offs between logic consistency and accuracy.
  arXiv  Detail & Related papers  (2026-05-19T07:27:50Z)
• Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens [4.0704009036918025]
  Reasoning in Large Language Models incurs significant inference-time compute.<n>We show that reasoning tokens split into two functional types: low-entropy textitstructural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy textitorganic tokens (problem-specific content that drives toward a solution)<n>We derive textitsupertokens that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-04-29T07:06:43Z)
• Large Language Models Decide Early and Explain Later [47.20810161393936]
  We study the evolution of predicted answers over reasoning steps using forced answer completion.<n>We show that simples, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query.
  arXiv  Detail & Related papers  (2026-04-24T06:26:24Z)
• SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio [0.038379177968040606]
  We propose SELFDOUBT, a single-pass uncertainty framework for reasoning language models.<n>Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior.<n> SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API.
  arXiv  Detail & Related papers  (2026-04-07T19:19:29Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit [114.83867400179354]
  Overthinking can degrade overall performance of large language models.<n>We categorize reasoning into three stages: insufficient exploration stage, compensatory reasoning stage, and reasoning convergence stage.<n>We develop a lightweight thresholding strategy based on rules to improve reasoning accuracy.
  arXiv  Detail & Related papers  (2025-08-25T03:17:17Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.931194824519935]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.