Expected Improvement via Gradient Norms

• URL: http://arxiv.org/abs/2601.21357v1
• Date: Thu, 29 Jan 2026 07:37:13 GMT
• Title: Expected Improvement via Gradient Norms
• Abstract summary: Expected Improvement (EI) is known to be overly exploitative and can converge to suboptimal stationary points.<n>We propose Expected Improvement via Gradient Norms (EI-GN), a novel acquisition function that applies the improvement principle to a gradient-aware auxiliary objective.<n>EI-GN relies on gradient observations used to learn gradient-enhanced surrogate models that enable principled gradient inference from function evaluations.
• Score: 2.542957206389996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian Optimization (BO) is a principled approach for optimizing expensive black-box functions, with Expected Improvement (EI) being one of the most widely used acquisition functions. Despite its empirical success, EI is known to be overly exploitative and can converge to suboptimal stationary points. We propose Expected Improvement via Gradient Norms (EI-GN), a novel acquisition function that applies the improvement principle to a gradient-aware auxiliary objective, thereby promoting sampling in regions that are both high-performing and approaching first-order stationarity. EI-GN relies on gradient observations used to learn gradient-enhanced surrogate models that enable principled gradient inference from function evaluations. We derive a tractable closed-form expression for EI-GN that allows efficient optimization and show that the proposed acquisition is consistent with the improvement-based acquisition framework. Empirical evaluations on standard BO benchmarks demonstrate that EI-GN yields consistent improvements against standard baselines. We further demonstrate applicability of EI-GN to control policy learning problems.

Related papers

• Policy Improvement Reinforcement Learning [40.05196753615896]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models.<n>Existing methods share a common blind spot: they optimize policies based on instantaneous group-level or batch-level statistics without ever verifying whether the resulting update actually improved the model.<n>We argue that the missing ingredient is policy improvement feedback: the ability to measure and optimize inter-iteration progress directly.
  arXiv  Detail & Related papers  (2026-04-01T13:10:20Z)
• Gradient Regularized Natural Gradients [20.85716825925689]
  We propose a family of scalable second-order gradients that integrate explicit regularization with natural gradient updates.<n>We establish convergence guarantees for GRNG, showing that gradient regularization improves stability and enables convergence to global minima.
  arXiv  Detail & Related papers  (2026-01-26T12:25:04Z)
• AWPO: Enhancing Tool-Use of Large Language Models through Explicit Integration of Reasoning Rewards [60.2998874976509]
  We propose advantage-weighted policy optimization (AWPO) to integrate explicit reasoning rewards to enhance tool-use capability.<n>AWPO incorporates variance-aware gating and difficulty-aware weighting to adaptively modulate advantages from reasoning signals.<n>Experiments demonstrate that AWPO achieves state-of-the-art performance across standard tool-use benchmarks.
  arXiv  Detail & Related papers  (2025-12-22T08:07:00Z)
• OBLR-PO: A Theoretical Framework for Stable Reinforcement Learning [12.77713716713937]
  We provide a unified theoretical framework that characterizes the statistical properties of commonly used policy-gradient estimators.<n>We derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of gradients.<n>We further show that the variance-optimal baseline is a gradient-weighted estimator, offering a new principle for variance reduction.
  arXiv  Detail & Related papers  (2025-11-28T16:09:28Z)
• Latent Chain-of-Thought for Visual Reasoning [53.541579327424046]
  Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs)<n>We reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference.<n>We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-27T23:10:06Z)
• OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessment [55.59322229889159]
  We propose OmniQuality-R, a unified reward modeling framework that transforms multi-task quality reasoning into continuous and interpretable reward signals.<n>We use a reasoning-enhanced reward modeling dataset to form a reliable chain-of-thought dataset for supervised fine-tuning.<n>We evaluate OmniQuality-R on three key IQA tasks: aesthetic quality assessment, technical quality evaluation, and text-image alignment.
  arXiv  Detail & Related papers  (2025-10-12T13:46:28Z)
• LARES: Latent Reasoning for Sequential Recommendation [96.26996622771593]
  We present LARES, a novel and scalable LAtent REasoning framework for Sequential recommendation.<n>Our proposed approach employs a recurrent architecture that allows flexible expansion of reasoning depth without increasing parameter complexity.<n>Our framework exhibits seamless compatibility with existing advanced models, further improving their recommendation performance.
  arXiv  Detail & Related papers  (2025-05-22T16:22:54Z)
• A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization [6.745285924097932]
  We introduce a unified theoretical framework, Variational Entropy Search, which reveals that EI and information-theoretic acquisition functions are more closely related than previously recognized.<n>We propose VES-Gamma, a novel acquisition function that balances the strengths of EI and MES.
  arXiv  Detail & Related papers  (2025-01-30T21:15:00Z)
• Unexpected Improvements to Expected Improvement for Bayesian Optimization [21.901803477674264]
  We propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically.<n>Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions.
  arXiv  Detail & Related papers  (2023-10-31T17:59:56Z)
• Adjusted Expected Improvement for Cumulative Regret Minimization in Noisy Bayesian Optimization [37.99915557191065]
  We adapt the expected improvement function for better performance under cumulative regret. A novel quantity called the evaluation cost is compared against the acquisition function. We establish a high-probability regret upper bound of EIC based on the maximum information gain.
  arXiv  Detail & Related papers  (2022-05-10T13:50:10Z)
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes [70.80716221080118]
  The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning. The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
  arXiv  Detail & Related papers  (2021-12-01T23:15:09Z)
• Zeroth-Order Hybrid Gradient Descent: Towards A Principled Black-Box Optimization Framework [100.36569795440889]
  This work is on the iteration of zero-th-order (ZO) optimization which does not require first-order information. We show that with a graceful design in coordinate importance sampling, the proposed ZO optimization method is efficient both in terms of complexity as well as as function query cost.
  arXiv  Detail & Related papers  (2020-12-21T17:29:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.