Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2607.23605v1
• Date: Sun, 26 Jul 2026 11:16:59 GMT
• Title: Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
• Abstract summary: HyGAE is an actor-critic framework that jointly optimize token- and turn-level objectives with the hybrid advantage and unified critic.<n>It achieves an average success rate of 91% and a significant improvement of 10% over other methods.
• Score: 69.23931086069983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (VLMs) now act as agents in interactive environments, where success requires coherent reasoning and decision-making across turns. Although end-to-end training in agentic environments can improve such multi-turn decision-making abilities, current methods mainly rely on either token-wise optimization over concatenated token trajectories or turn-wise optimization with uniform within-turn credit. In this work, we establish theoretical formulations for the two levels of optimization and derive a hybrid advantage that serves both objectives. Furthermore, with an appropriate choice of discount factor and learning target, we prove that a unified critic model can estimate values for both turn-wise and token-wise. As such, we propose HyGAE, an actor-critic framework that jointly optimizes token- and turn-level objectives with the hybrid advantage and unified critic. We conduct extensive evaluations of HyGAE across five multi-turn decision-making environments, where it achieves an average success rate of 91% and a significant improvement of 10% over other methods. Furthermore, we provide an in-depth analysis showing that the exact analytic form of the hybrid advantage and return is crucial for optimization. Project Page: https://wx-zhang.github.io/hygae-web/.

Related papers

• Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity [13.065036862403966]
  This paper proposes a unified framework, KENDO, that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies.<n>KENDO-BO achieves competitive or superior optimization performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-08-25T15:35:47Z)
• Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization [54.855135506924775]
  We introduce a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives.<n>This design enables fine-grained, action-level optimization across multiple conflicting objectives.
  arXiv  Detail & Related papers  (2026-06-15T01:58:07Z)
• Pragmatic Curiosity: A Hybrid Learning-Optimization Paradigm via Active Inference [20.135421015458817]
  "pragmatic curiosity" is a hybrid learning-optimization paradigm derived from active inference.<n>We demonstrate the practical effectiveness and flexibility of pragmatic curiosity on various real-world hybrid tasks.
  arXiv  Detail & Related papers  (2026-02-05T18:42:29Z)
• Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning [17.84264663466905]
  Surrogate-assisted Evolutionary Algorithms(SAEAs) have shown promising robustness in solving expensive optimization problems.<n>A key aspect that impacts SAEAs' effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer.<n>We propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process.
  arXiv  Detail & Related papers  (2026-01-31T06:14:27Z)
• Optimizers Qualitatively Alter Solutions And We Should Leverage This [62.662640460717476]
  Deep Neural Networks (DNNs) can not guarantee convergence to a unique global minimum of the loss when using only local information, such as SGD.<n>We argue that the community should aim at understanding the biases of already existing methods, as well as aim to build new DNNs with the explicit intent of inducing certain properties of the solution.
  arXiv  Detail & Related papers  (2025-07-16T13:33:31Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Preference-Guided Diffusion for Multi-Objective Offline Optimization [64.08326521234228]
  We propose a preference-guided diffusion model for offline multi-objective optimization.<n>Our guidance is a preference model trained to predict the probability that one design dominates another.<n>Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions.
  arXiv  Detail & Related papers  (2025-03-21T16:49:38Z)
• MOHPER: Multi-objective Hyperparameter Optimization Framework for E-commerce Retrieval System [1.4323566945483497]
  MOHPER is a multi-objective optimization framework for e-commerce sites.<n>It jointly optimize click-through rate (CTR), click-through conversion rate (CTCVR) and relevant objectives.<n>It substantiates its practical efficacy in achieving a balanced optimization that aligns with both user satisfaction and revenue goals.
  arXiv  Detail & Related papers  (2025-03-07T08:25:08Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)
• Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal [0.0]
  We propose a general classification of adaptive sampling techniques to highlight similarities and differences between the vast families of adaptive sampling, active learning, and Bayesian optimization. We provide guidelines to apply those learning criteria investigating the performance of different Bayesian schemes for a variety of benchmark problems.
  arXiv  Detail & Related papers  (2023-03-02T20:22:40Z)
• Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training [52.39882976848064]
  We design a Generalized Krasnoselskii-Mann (GKM) scheme based on fixed-point iterations as our fundamental ODL module. Under the GKM scheme, a Bilevel Meta Optimization (BMO) algorithmic framework is constructed to solve the optimal training and hyper-training variables together.
  arXiv  Detail & Related papers  (2022-06-16T01:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.