Is High Variance Unavoidable in RL? A Case Study in Continuous Control

• URL: http://arxiv.org/abs/2110.11222v1
• Date: Thu, 21 Oct 2021 15:59:37 GMT
• Title: Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Abstract summary: Reinforcement learning experiments have notoriously high variance. Minor details can have disproportionately large effects on measured outcomes. We show that one cause for early variance is numerical instability.
• Score: 42.960199987696306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle for real-world applications, where safety and predictability are paramount. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance -- continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that variance mostly arises early in training as a result of poor "outlier" runs, but that weight initialization and initial exploration are not to blame. We show that one cause for early variance is numerical instability which leads to saturating nonlinearities. We investigate several fixes to this issue and find that one particular method is surprisingly effective and simple -- normalizing penultimate features. Addressing the learning instability allows for larger learning rates, and significantly decreases the variance of outcomes. This demonstrates that the perceived variance in RL is not necessarily inherent to the problem definition and may be addressed through simple architectural modifications.

Related papers

• Demystifying Variance in Circuit Discovery of LLMs [51.97489679448601]
  We develop a new circuit discovery method that substantially lessens resampling variance.<n>We show that rephrasing variance arises because prompts with different templates tend to activate different circuits in the model.<n>We argue that sparsity, which has been claimed to form more compact and interpretable task circuits, fails to solve this problem.
  arXiv  Detail & Related papers  (2026-06-15T16:25:07Z)
• Output Embedding Centering for Stable LLM Pretraining [3.0718163016168067]
  A specific instability that often occurs for large learning rates at the end of training is output logit divergence.<n>We propose output embedding centering (OEC) as a new mitigation strategy.<n>OEC can be implemented in two different ways, as a deterministic operation called -centering, or a regularization method called -loss.
  arXiv  Detail & Related papers  (2026-01-05T11:44:05Z)
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning [23.202550214006237]
  It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to textitdiversity collapse, where outputs lack variety.<n>We introduce a principled method -- textitdifferential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based languages.
  arXiv  Detail & Related papers  (2025-11-25T05:28:55Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Revisiting Essential and Nonessential Settings of Evidential Deep Learning [70.82728812001807]
  Evidential Deep Learning (EDL) is an emerging method for uncertainty estimation. We propose Re-EDL, a simplified yet more effective variant of EDL.
  arXiv  Detail & Related papers  (2024-10-01T04:27:07Z)
• Multiple Descents in Unsupervised Learning: The Role of Noise, Domain Shift and Anomalies [14.399035468023161]
  We study the presence of double descent in unsupervised learning, an area that has received little attention and is not yet fully understood. We use synthetic and real data and identify model-wise, epoch-wise, and sample-wise double descent for various applications.
  arXiv  Detail & Related papers  (2024-06-17T16:24:23Z)
• Dissecting Deep RL with High Update Ratios: Combatting Value Divergence [21.282292112642747]
  We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters. We employ a simple unit-ball normalization that enables learning under large update ratios.
  arXiv  Detail & Related papers  (2024-03-09T19:56:40Z)
• Variance of ML-based software fault predictors: are we really improving fault prediction? [0.3222802562733786]
  We experimentally analyze the variance of a state-of-the-art fault prediction approach. We observed a maximum variance of 10.10% in terms of the per-class accuracy metric.
  arXiv  Detail & Related papers  (2023-10-26T09:31:32Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• Detecting Rewards Deterioration in Episodic Reinforcement Learning [63.49923393311052]
  In many RL applications, once training ends, it is vital to detect any deterioration in the agent performance as soon as possible. We consider an episodic framework, where the rewards within each episode are not independent, nor identically-distributed, nor Markov. We define the mean-shift in a way corresponding to deterioration of a temporal signal (such as the rewards), and derive a test for this problem with optimal statistical power.
  arXiv  Detail & Related papers  (2020-10-22T12:45:55Z)
• What causes the test error? Going beyond bias-variance via ANOVA [21.359033212191218]
  Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. We propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way.
  arXiv  Detail & Related papers  (2020-10-11T05:21:13Z)
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction [96.90215318875859]
  We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from corrective feedback. We propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training.
  arXiv  Detail & Related papers  (2020-03-16T16:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.