The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection

• URL: http://arxiv.org/abs/2605.26872v1
• Date: Tue, 26 May 2026 11:30:17 GMT
• Title: The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection
• Abstract summary: Student-Centric Answer Sampling (SCAS) is a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost.<n> Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance.
• Score: 46.38575272632866
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.

Related papers

• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold [4.090731563364332]
  We argue that reinforcement learning policies should be viewed as local probes of a multi-basin reasoning solution manifold.<n>We propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation.
  arXiv  Detail & Related papers  (2026-07-30T07:05:37Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Masking Teacher and Reinforcing Student for Distilling Vision-Language Models [50.619420197124356]
  Large-scale vision-language models (VLMs) have recently achieved remarkable multimodal understanding.<n>This raises the need for compact yet capable VLMs that can efficiently learn from powerful large teachers.<n>We propose Masters (Masking Teacher and Reinforcing Student), a mask-progressive reinforcement learning framework.
  arXiv  Detail & Related papers  (2025-12-23T14:40:38Z)
• In Good GRACEs: Principled Teacher Selection for Knowledge Distillation [32.00881805808886]
  We propose a lightweight score called GRACE to quantify how effective a teacher will be for post-training a student model.<n> GRACE measures distributional properties of the student's gradients without access to a verifier, teacher logits, teacher internals, or test data.
  arXiv  Detail & Related papers  (2025-11-04T18:58:47Z)
• Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data [18.97748910748554]
  We introduce Local Naturalness, which measures the student's log-probabilities over short, sequential reasoning steps conditioned only on a small local window.<n>When mixing answers from many teachers, Local Naturalness boosts a 32B student's accuracy on math benchmarks by 9.4pp over global selection.<n>These results highlight the power of localized data quality evaluation and data mixing for more effective reasoning distillation.
  arXiv  Detail & Related papers  (2025-10-05T01:15:32Z)
• Merge-of-Thought Distillation [23.53356244978525]
  Merge-of-Thought Distillation (MoT) is a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging the resulting student variants.<n>On competition math benchmarks, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1.<n>MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics, and shows robustness to distribution-shifted and peer-level teachers.
  arXiv  Detail & Related papers  (2025-09-10T17:46:57Z)
• Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization [84.86241161706911]
  We show that teacher LLMs can indeed intervene on student reasoning to improve their performance. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learn from explained data. We verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.
  arXiv  Detail & Related papers  (2023-06-15T17:27:20Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• Faculty Distillation with Optimal Transport [53.69235109551099]
  We propose to link teacher's task and student's task by optimal transport. Based on the semantic relationship between their label spaces, we can bridge the support gap between output distributions. Experiments under various settings demonstrate the succinctness and versatility of our method.
  arXiv  Detail & Related papers  (2022-04-25T09:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.