Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data

• URL: http://arxiv.org/abs/2510.03988v1
• Date: Sun, 05 Oct 2025 01:15:32 GMT
• Title: Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data
• Abstract summary: We introduce Local Naturalness, which measures the student's log-probabilities over short, sequential reasoning steps conditioned only on a small local window.<n>When mixing answers from many teachers, Local Naturalness boosts a 32B student's accuracy on math benchmarks by 9.4pp over global selection.<n>These results highlight the power of localized data quality evaluation and data mixing for more effective reasoning distillation.
• Score: 18.97748910748554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distilling long reasoning traces (10K+ tokens) from stronger teacher models into smaller student LLMs via SFT has emerged as a standard paradigm. This approach is practical and efficient: it leverages the ease of generating abundant reasoning data from stronger models and provides a direct, data-driven way to teach less capable models better reasoning. While previous work has largely focused on prompt selection with responses from a single teacher, the equally important problem of choosing the best response when multiple teacher outputs are available for a single prompt remains underexplored. This challenge becomes important in a multi-teacher setting, where different students may benefit from the outputs of different teachers. This paper fills that gap with a systematic study of response selection for reasoning distillation. We first show that the current method, which picks responses the student assigns the highest global log-probability (global naturalness), fails when responses come from multiple teachers, i.e., global naturalness no longer correlates with downstream performance, especially as the reasoning traces from strong teachers become longer. To overcome this problem, we introduce Local Naturalness, which measures the student's log-probabilities over short, sequential reasoning steps conditioned only on a small local window. Local Naturalness enables two applications: 1) Teacher Selection: Aggregating local scores across prompts reliably identifies the most helpful teacher. 2) Response Selection from a Multiple Teachers: When mixing answers from many teachers, Local Naturalness boosts a 32B student's accuracy on math benchmarks by 9.4pp over global selection, also surpassing the performance achieved by training on data from the single best teacher. These results highlight the power of localized data quality evaluation and data mixing for more effective reasoning distillation.

Related papers

• Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models [68.52262705081098]
  On-policy distillation transfers teacher capabilities by supervising trajectories sampled from the student's own policy.<n>We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems.
  arXiv  Detail & Related papers  (2026-08-17T14:46:53Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection [46.38575272632866]
  Student-Centric Answer Sampling (SCAS) is a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost.<n> Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance.
  arXiv  Detail & Related papers  (2026-05-26T11:30:17Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment [82.00769536768509]
  Rank-Surprisal Ratio is a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory.<n>We demonstrate its practical utility in both trajectory selection and teacher selection.
  arXiv  Detail & Related papers  (2026-01-20T18:58:10Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration [103.1589018460702]
  "guidance-on-demand" approach expands exploration while preserving the value of self-discovery.<n>Experiments show AMPO substantially outperforms a strong baseline.<n>Using four peer-sized teachers, our method achieves comparable results to approaches that leverage a single, more powerful teacher.
  arXiv  Detail & Related papers  (2025-10-02T17:14:00Z)
• Merge-of-Thought Distillation [23.53356244978525]
  Merge-of-Thought Distillation (MoT) is a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging the resulting student variants.<n>On competition math benchmarks, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1.<n>MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics, and shows robustness to distribution-shifted and peer-level teachers.
  arXiv  Detail & Related papers  (2025-09-10T17:46:57Z)
• NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks [65.70224757972068]
  We select reasoning traces from a strong teacher model based on a large pool of questions from NaturalReasoning.<n>We find that simply scaling up data size with random sampling is a strong baseline with steady performance gains.<n>We find that selecting difficult examples that require more diverse reasoning strategies is more sample-efficient to transfer the teacher model's reasoning skills.
  arXiv  Detail & Related papers  (2025-07-02T17:30:24Z)
• Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review [11.756344944226495]
  We introduce a novel Fault-Aware DistIllation via Peer-Review (FAIR) approach.<n>Instead of merely obtaining rationales from teachers, our method asks teachers to identify and explain the student's mistakes.<n>Our method reduces the chance of teachers guessing correctly with flawed rationale.
  arXiv  Detail & Related papers  (2024-10-04T17:59:41Z)
• Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracy [0.0]
  This paper introduces AMMORE, a new dataset of 53,000 math open-response question-answer pairs from Rori. We conduct two experiments to evaluate the use of large language models (LLM) for grading challenging student answers.
  arXiv  Detail & Related papers  (2024-09-26T14:51:40Z)
• Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios [3.818273633647809]
  We propose a three-component framework leveraging three signal types. The first signal is the student's self-consistency (consistency of student multiple outputs), which is a proxy of the student's confidence. We show that our proposed two-stage framework brings a relative improvement of up to 20.79% compared to fine-tuning without any signals across datasets.
  arXiv  Detail & Related papers  (2024-06-08T02:17:43Z)
• Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization [84.86241161706911]
  We show that teacher LLMs can indeed intervene on student reasoning to improve their performance. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learn from explained data. We verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.
  arXiv  Detail & Related papers  (2023-06-15T17:27:20Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Faculty Distillation with Optimal Transport [53.69235109551099]
  We propose to link teacher's task and student's task by optimal transport. Based on the semantic relationship between their label spaces, we can bridge the support gap between output distributions. Experiments under various settings demonstrate the succinctness and versatility of our method.
  arXiv  Detail & Related papers  (2022-04-25T09:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.