ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

• URL: http://arxiv.org/abs/2608.03972v1
• Date: Tue, 04 Aug 2026 17:40:08 GMT
• Title: ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning
• Abstract summary: We show that flawed trajectories can still provide valuable reasoning signals when treated as flawed trajectories to reflect upon.<n>We propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training.
• Score: 84.29802893743991
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.

Related papers

• Revisiting Complete Reasoning Traces for Post-Training [61.76889440384448]
  Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability.<n>We show that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation.
  arXiv  Detail & Related papers  (2026-09-07T06:46:17Z)
• LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation [25.555783105371574]
  We propose a training framework that formulates reflection as a memory-control policy.<n>LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism.<n> Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-08-12T11:56:03Z)
• Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR [30.94808389410323]
  Trajectory-Guided Reinforcement Learning guides the policy model to integrate visual evidence into fine-grained reasoning processes.<n>Experiments on multiple multimodal reasoning benchmarks demonstrate that TGRL consistently improves reasoning performance.
  arXiv  Detail & Related papers  (2026-03-27T07:18:18Z)
• Learn Hard Problems During RL with Reference Guided Fine-tuning [56.56461712665904]
  Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity.<n>We introduce Reference-Guided Fine-Tuning (ReGFT) to synthesize positive trajectories on hard problems and train on them before RL.<n>Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
  arXiv  Detail & Related papers  (2026-03-01T18:41:28Z)
• Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing [70.35701681177655]
  Self-improvement has emerged as a mainstream paradigm for advancing the reasoning capabilities of large vision-language models.<n>We introduce four efficient strategies to achieve head-tail re-balancing during the exploration-and-learning self-improvement process.<n>Our methods consistently improve visual reasoning capabilities, outperforming vanilla self-improvement by 3.86 points on average.
  arXiv  Detail & Related papers  (2025-10-30T13:26:58Z)
• First Try Matters: Revisiting the Role of Reflection in Reasoning Models [66.39546876232512]
  We focus on reflective behaviours where the model has already produced an answer but continues reflecting before finalizing its output.<n>Our analysis reveals that reflections are predominantly confirmatory and rarely alter the model's initial answer.<n>We propose a question-aware early-stopping method that enhances inference-time token efficiency by stopping the reasoning process once a few plausible candidate answers are generated.
  arXiv  Detail & Related papers  (2025-10-09T14:57:10Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Learning to Reason under Off-Policy Guidance [40.27817638425237]
  We introduce textbfLUFFY (textbfLearning to reason textbfUnder otextbfFF-polictextbfY guidance), a framework that augments textitRLVR with off-policy reasoning traces.<n>LUFFY balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training.
  arXiv  Detail & Related papers  (2025-04-21T08:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.