Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher

• URL: http://arxiv.org/abs/2606.25407v1
• Date: Wed, 24 Jun 2026 05:07:41 GMT
• Title: Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher
• Abstract summary: Chest X-ray visual question answering (CXR VQA) requires models not only to predict correct answers, but also to produce reliable medical reasoning.<n>We propose text-to-Reason (T2R), a framework that introduces comparison-based supervision into CoT optimization.
• Score: 18.162030833167385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chest X-ray visual question answering (CXR VQA) requires models not only to predict correct answers, but also to produce reliable medical reasoning. However, existing reinforcement-learning-based training typically relies on answer-level rewards, which are often too coarse to improve chain-of-thought (CoT) quality and can become ineffective when group-level advantages collapse to zero. We propose \textbf{Teach-to-Reason (T2R)}, a framework that introduces comparison-based supervision into CoT optimization through a self-improving \emph{Teacher} and a competition-guided \emph{Reasoner}. As the Teacher is iteratively strengthened via self-competition, the Reasoner is optimized against progressively stronger Teacher-generated references. We further introduce a case-wise reward design that preserves the original reward-induced positive/negative partition when it is informative, and restores supervision from competition scores when the original reward signal degenerates. Experiments on multiple CXR open-ended VQA benchmarks show that T2R consistently outperforms strong baselines, indicating that comparison-based supervision, when integrated in a controlled and principled manner, provides a more effective training signal for reasoning optimization.

Related papers

• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions [80.33276868889341]
  Z-Reward decouples reasoning-heavy judgment from efficient reward deployment.<n>We show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization.
  arXiv  Detail & Related papers  (2026-06-08T06:20:12Z)
• RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time [51.256131853751754]
  We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
  arXiv  Detail & Related papers  (2026-04-13T15:38:09Z)
• P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering [51.04492568024515]
  We introduce Probabilistic Process Supervision (P2S), a novel framework for fine-grained process rewards.<n>P2S provides fine-grained process rewards without requiring a separate reward model or human-annotated reasoning steps.
  arXiv  Detail & Related papers  (2026-01-28T14:35:20Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models [33.398631680508814]
  We propose Answer-Consistent Reinforcement Learning that modifies the GRPO algorithm with an auxiliary consistency check.<n>We design a consistency-verification reward that grants a high reward only if both the original and the post-shuffle answers agree and are correct.<n>We evaluate ACRE on challenging Video Reasoning benchmarks and multimodal math reasoning benchmarks, achieving an average 2.2% and 1.5% improvement.
  arXiv  Detail & Related papers  (2025-10-11T08:32:52Z)
• Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense [36.71358559780692]
  HERO is a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way.<n> HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks.
  arXiv  Detail & Related papers  (2025-10-08T17:09:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.