Related papers: Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

URL: http://arxiv.org/abs/2607.14552v1
Date: Thu, 16 Jul 2026 04:28:19 GMT
Title: Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
Abstract summary: We show that a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer.<n>We show that this second step degrades the training data in a way that correctness filtering cannot catch.
Score: 21.049419375153914
License: http://creativecommons.org/licenses/by/4.0/
Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors. When sampling fails, a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer. We show that this second step degrades the training data in a way that correctness filtering cannot catch. We run a controlled experiment that fixes the generator, the problem set, and the correctness filter, and varies only whether the chain is generated under answer-conditioning, the gold answer shown with a request to reach it. Training a strong instruction-tuned reasoning model on its own answer-conditioned chains sharply lowers its verifiable-reasoning accuracy. The loss grows with difficulty, reaching as much as about 27 points on the hardest competition problems. The mechanism is legible in the chains themselves, which rationalize backward from the shown answer instead of deriving it, with the early final-answer statement as the measurable symptom. The harm is a property of the data rather than the generator, read off unlabeled generations before any fine-tuning, ordering the penalty across eight thinking models from four families, and transferring across teacher families. A prompt ablation localizes it to the rationalize-toward instruction rather than the answer's bare visibility. The practical takeaway is to generate answer-blind, because no correctness filter can see this damage in the data.

Related papers

Consensus as Privileged Context for Label-Free Self-Distillation [12.523025093871304]
CANON (Consensus-ANchored self-distillatiON) is a label-free training method that turns consensus into dense, token-level supervision.<n>CanON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute.
arXiv Detail & Related papers (2026-07-15T09:39:32Z)
Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA [40.94400211806987]
Language models are increasingly taught from synthetic question--answer (QA) supervision.<n>We show that this generation step is not neutral preprocessing.<n>We show that this failure modes arise from choices made during QA generation, so they can be reduced without changing the training loop.
arXiv Detail & Related papers (2026-06-30T17:35:14Z)
CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution [52.691495954442985]
CoVerRL is a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other.<n> Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks.<n>Self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
arXiv Detail & Related papers (2026-03-18T14:38:55Z)
Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models [33.398631680508814]
We propose Answer-Consistent Reinforcement Learning that modifies the GRPO algorithm with an auxiliary consistency check.<n>We design a consistency-verification reward that grants a high reward only if both the original and the post-shuffle answers agree and are correct.<n>We evaluate ACRE on challenging Video Reasoning benchmarks and multimodal math reasoning benchmarks, achieving an average 2.2% and 1.5% improvement.
arXiv Detail & Related papers (2025-10-11T08:32:52Z)
Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation [25.195244084313114]
Chain-of-Thought Correctness Perception Distillation (CoPeD) aims to improve the reasoning quality of the student model.<n>CoPeD encourages the student model to predict answers based on correct rationales and revise them when they are incorrect.
arXiv Detail & Related papers (2025-09-06T05:33:17Z)
Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think [51.0691253204425]
We analyze intermediate reasoning steps, termed subthoughts, to answer two questions: Does the final answer reliably represent the model's optimal conclusion?<n>Our approach involves segmenting a reasoning trace into sequential subthoughts based on linguistic cues.<n>We find that aggregating these answers by selecting the most frequent one (the mode) often yields significantly higher accuracy compared to relying solely on the answer derived from the original complete trace.
arXiv Detail & Related papers (2025-04-29T12:39:07Z)
Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step [81.50681925980135]
We propose a method to probe changes in the mind during the model's reasoning.<n>By analyzing patterns in mind change, we examine the correctness of the model's reasoning.<n>Our validation reveals that many responses, although correct in their final answer, contain errors in their reasoning process.
arXiv Detail & Related papers (2024-06-23T15:50:22Z)
Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering [59.495717939664246]
Large language models have manifested remarkable capabilities by leveraging chain-of-thought (CoT) reasoning techniques to solve intricate questions. We propose a novel approach called the selective filtering reasoner (SelF-Reasoner) that assesses the entailment relationship between the question and the candidate reasoning chain. SelF-Reasoner improves the fine-tuned T5 baseline consistently over the ScienceQA, ECQA, and LastLetter tasks.
arXiv Detail & Related papers (2024-03-28T06:28:35Z)
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step [133.60124577507727]
Chain-of-thought prompting primes large language models to verbalize rationalization for their predictions. We show that orders-of-magnitude smaller models can still benefit from chain-of-thought prompting. We introduce Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.
arXiv Detail & Related papers (2023-06-24T20:15:07Z)
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning [26.715242799194908]
We show how a question-answering system can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Our approach is to combine a trained backward-chaining model, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system's own internal beliefs)
arXiv Detail & Related papers (2022-10-21T19:51:56Z)
Measuring and Narrowing the Compositionality Gap in Language Models [116.5228850227024]
We measure how often models can correctly answer all sub-problems but not generate the overall solution. We present a new method, self-ask, that further improves on chain of thought.
arXiv Detail & Related papers (2022-10-07T06:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.