Teaching Large Reasoning Models Effective Reflection

• URL: http://arxiv.org/abs/2601.12720v1
• Date: Mon, 19 Jan 2026 04:51:53 GMT
• Title: Teaching Large Reasoning Models Effective Reflection
• Authors: Hanbin Wang, Jingwei Song, Jinpeng Li, Qi Zhu, Fei Mi, Ganqu Cui, Yasheng Wang, Lifeng Shang,
• Abstract summary: Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks.<n>However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer.<n>We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques.
• Score: 62.73646680747003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.

Related papers

• OCR-Agent: Agentic OCR with Capability and Memory Reflection [5.8505408398110434]
  Large Vision-Language Models (VLMs) have demonstrated significant potential on complex visual understanding tasks.<n>We propose a novel iterative self-correction framework that endows models with two key capabilities: Capability Reflection and Memory Reflection.<n> Experiments on the challenging OCRBench v2 benchmark show that OCR-Agent outperforms the current open-source SOTA model InternVL3-8B by +2.0 on English and +1.2 on Chinese subsets.
  arXiv  Detail & Related papers  (2026-02-24T16:10:27Z)
• Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning [89.60378227969643]
  We propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision.<n>Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly.<n>Experiments across various tasks and models show that Critique-RL delivers substantial performance improvements.
  arXiv  Detail & Related papers  (2025-10-28T11:37:01Z)
• First Try Matters: Revisiting the Role of Reflection in Reasoning Models [66.39546876232512]
  We focus on reflective behaviours where the model has already produced an answer but continues reflecting before finalizing its output.<n>Our analysis reveals that reflections are predominantly confirmatory and rarely alter the model's initial answer.<n>We propose a question-aware early-stopping method that enhances inference-time token efficiency by stopping the reasoning process once a few plausible candidate answers are generated.
  arXiv  Detail & Related papers  (2025-10-09T14:57:10Z)
• RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback [57.967762383794806]
  RefCritic is a long-chain-of-thought critic module based on reinforcement learning with dual rule-based rewards.<n>We evaluate RefCritic on Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B across five benchmarks.
  arXiv  Detail & Related papers  (2025-07-20T16:19:51Z)
• Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Models [29.615519143908998]
  Self-affirmation reflections are redundant reflective steps that affirm prior content and often occur after the already correct reasoning steps.<n>We show that suppressing self-affirmation reflections reduces output length without degrading accuracy across multiple models.<n>We also improve current train-based method by explicitly suppressing such reflections.
  arXiv  Detail & Related papers  (2025-06-14T05:30:09Z)
• Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction [11.838351314880736]
  Instruct-of-Reflection (IoRT) is a novel and general reflection framework that leverages dynamic-meta instruction to enhance the iterative reflection capability of Large Language Models (LLMs)<n>Our experiments demonstrate that IoRT achieves an average improvement of 10.1% over established baselines in mathematical and commonsense reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-02T14:02:03Z)
• Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time [22.397964526411812]
  We propose a Dual-model Reflective Scoring framework featuring a dedicated Critic model trained for effective reflection.<n>DARS achieves strong performance and consistently outperforms existing ASAS baselines across all evaluation metrics.
  arXiv  Detail & Related papers  (2025-02-26T15:41:41Z)
• RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques [59.861013614500024]
  We introduce a new benchmark designed to assess the critique capabilities of Large Language Models (LLMs)<n>Unlike existing benchmarks, which typically function in an open-loop fashion, our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques.
  arXiv  Detail & Related papers  (2025-01-24T13:48:10Z)
• Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision [120.40788744292739]
  We propose a two-player paradigm that separates the roles of reasoning and critique models. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time.
  arXiv  Detail & Related papers  (2024-11-25T17:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.