Related papers: Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

URL: http://arxiv.org/abs/2607.13274v1
Date: Tue, 14 Jul 2026 21:20:42 GMT
Title: Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Abstract summary: Reinforcement learning is increasingly being considered for controlling real-world systems.<n>Actor-critic algorithms share a set of design decisions, such as how the policy is updated.<n>We analyze over 33,000 experiments to determine how these components affect variability across runs.
Score: 20.80238495293659
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.

Related papers

Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
arXiv Detail & Related papers (2026-07-29T05:16:59Z)
Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization [0.0]
Geosteering requires navigating a well trajectory through an unknown geological configuration.<n>This work presents an uncertainty-aware geosteering framework that integrates particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning for sequential decision-making.
arXiv Detail & Related papers (2026-06-15T22:22:13Z)
Uncertainty Estimation for Deep Reconstruction in Actuatic Disaster Scenarios with Autonomous Vehicles [68.82821445619759]
principled uncertainty quantification is critical for active sensing strategies.<n>Evidential Deep Learning achieves the best reconstruction accuracy and uncertainty calibration across all sensor configurations.<n>These findings support Evidential Deep Learning as the preferred method for uncertainty-aware field reconstruction in real-time autonomous vehicle deployments.
arXiv Detail & Related papers (2026-04-07T19:17:44Z)
A Comedy of Estimators: On KL Regularization in RL Training of LLMs [81.7906270099878]
reinforcement learning (RL) can substantially improve the reasoning performance of large language models (LLMs)<n>The RL objective for LLM training involves a regularization term, which is the reverse Kullback-Leibler (KL) divergence between the trained policy and the reference policy.<n>Recent works show that prevailing practices for incorporating KL regularization do not provide correct gradients for stated objectives, creating a discrepancy between the objective and its implementation.<n>We study the gradients of several estimators configurations, revealing how design choices shape gradient bias.
arXiv Detail & Related papers (2025-12-26T04:20:58Z)
Decision-focused Sensing and Forecasting for Adaptive and Rapid Flood Response: An Implicit Learning Approach [4.746115078755406]
Timely and reliable decision-making is vital for flood emergency response.<n>Traditional flood management systems often rely on in-situ sensors to calibrate remote sensing-based large-scale flood depth forecasting models.<n>We introduce a novel decision-focused framework that strategically selects locations for in-situ sensor placement.
arXiv Detail & Related papers (2025-10-15T09:47:00Z)
Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
arXiv Detail & Related papers (2024-05-09T12:52:22Z)
Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
arXiv Detail & Related papers (2024-05-03T16:45:15Z)
Discovering Decision Manifolds to Assure Trusted Autonomous Systems [0.0]
We propose an optimization-based search technique for capturing the range of correct and incorrect responses a system could exhibit. This manifold provides a more detailed understanding of system reliability than traditional testing or Monte Carlo simulations. In this proof-of-concept, we apply our method to a software-in-the-loop evaluation of an autonomous vehicle.
arXiv Detail & Related papers (2024-02-12T16:55:58Z)
A Review of Uncertainty Calibration in Pretrained Object Detectors [5.440028715314566]
We investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation. We deliver novel insights into why poor detector calibration emerges.
arXiv Detail & Related papers (2022-10-06T14:06:36Z)
Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
arXiv Detail & Related papers (2022-03-14T17:40:42Z)
Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
arXiv Detail & Related papers (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.