FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

• URL: http://arxiv.org/abs/2605.20256v1
• Date: Mon, 18 May 2026 12:48:36 GMT
• Title: FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning
• Abstract summary: We propose FBOS-RL, a Feedback-Driven Bi-Objective Synergistic reinforcement learning framework.<n>Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment.<n>Under an identical number of rollouts, FBOS-RL learns substantially faster than GRPO and feedback-based baselines.
• Score: 16.200486964371713
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models. At its core, the training loop of GRPO and its variants alternates between rollout sampling and policy update. Unlike supervised learning, where each gradient step is anchored to an explicit ground-truth target, the optimal gradient direction for updating model parameters in this setting is not known a priori; the high-quality rollouts drawn during the sampling stage therefore act as the implicit "teacher" that guides every parameter update. However, GRPO adopt a simple sampling scheme that conditions all rollouts on the same original prompt. When a task lies beyond the policy model's current capability, this sampling scheme rarely yields a high-quality rollout, leaving the policy model without a meaningful gradient direction when updating its parameters, which causes training to stall. To address this issue, we propose FBOS-RL, a Feedback-Driven Bi-Objective Synergistic reinforcement learning framework. Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment, and on top of this we design two mutually reinforcing training objectives: Exploitation-oriented Policy Alignment(EPA) and Exploration-oriented Capability Cultivation(ECC). Extensive experiments demonstrate that EPA and ECC can mutually reinforce each other, forming a positive flywheel effect that significantly improves both the training efficiency and the final performance ceiling of reinforcement learning. Specifically, under an identical number of rollouts, FBOS-RL learns substantially faster than GRPO and feedback-based baselines and ultimately attains a higher performance ceiling, while exhibiting higher policy entropy and lower gradient norms throughout training.

Related papers

• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO [78.36537400975298]
  Group Relative Policy Optimization (GRPO) has demonstrated remarkable success in aligning text-to-image (T2I) flow models with human preferences.<n>We have identified that the learning loop of current flow-based GRPO is fundamentally decoupled from the learner's current capability.<n>We propose Adaptive GRPO (AdaGRPO), a novel capability-aware RL algorithm tailored for flow models.
  arXiv  Detail & Related papers  (2026-06-05T02:07:08Z)
• Post-Hoc Robustness for Model-Based Reinforcement Learning [1.4072254177584387]
  This work introduces post-hoc robustification of deep RL agents at inference time.<n>The goal is to improve robustness without any additional training of neural networks.
  arXiv  Detail & Related papers  (2026-06-02T11:43:13Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Poly-EPO: Training Exploratory Reasoning Models [62.82992914206963]
  We present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation.<n>We propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation.
  arXiv  Detail & Related papers  (2026-04-19T22:54:19Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning [44.07085022671951]
  Trajectories that introduce novel gradient directions receive a bounded multiplicative reward scaler.<n>G2RL consistently improves pass@1, maj@16, and pass@k over entropy based GRPO and external embedding methods.
  arXiv  Detail & Related papers  (2025-12-17T18:44:45Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoning [17.928214942495412]
  ACPO employs a dynamic curriculum that orchestrates a principled transition from a stable, near on-policy exploration phase to an efficient, off-policy exploitation phase.<n>We conduct extensive experiments on a suite of challenging multimodal reasoning benchmarks, including MathVista, LogicVista, and MMMU-Pro.<n>Results demonstrate that ACPO consistently outperforms strong baselines such as DAPO and PAPO, achieving state-of-the-art performance, accelerated convergence, and superior training stability.
  arXiv  Detail & Related papers  (2025-10-01T09:11:27Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO [22.00487909203855]
  Group Relative Policy Optimization fails to update a policy when all responses within a group are incorrect.<n>This limitation underscores a key gap between artificial and human intelligence.<n>We introduce a simple framework that mitigates the all-negative-sample issue by incorporating response diversity within groups.
  arXiv  Detail & Related papers  (2025-05-16T18:02:05Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.