Normalization Enhances Generalization in Visual Reinforcement Learning
- URL: http://arxiv.org/abs/2306.00656v1
- Date: Thu, 1 Jun 2023 13:24:56 GMT
- Title: Normalization Enhances Generalization in Visual Reinforcement Learning
- Authors: Lu Li, Jiafei Lyu, Guozheng Ma, Zilin Wang, Zhenjie Yang, Xiu Li,
Zhiheng Li
- Abstract summary: normalization techniques have demonstrated huge success in supervised and unsupervised learning.
We find that incorporating suitable normalization techniques is sufficient to enhance the generalization capabilities.
Our method significantly improves generalization capability while only marginally affecting sample efficiency.
- Score: 20.04754884180226
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in visual reinforcement learning (RL) have led to impressive
success in handling complex tasks. However, these methods have demonstrated
limited generalization capability to visual disturbances, which poses a
significant challenge for their real-world application and adaptability. Though
normalization techniques have demonstrated huge success in supervised and
unsupervised learning, their applications in visual RL are still scarce. In
this paper, we explore the potential benefits of integrating normalization into
visual RL methods with respect to generalization performance. We find that,
perhaps surprisingly, incorporating suitable normalization techniques is
sufficient to enhance the generalization capabilities, without any additional
special design. We utilize the combination of two normalization techniques,
CrossNorm and SelfNorm, for generalizable visual RL. Extensive experiments are
conducted on DMControl Generalization Benchmark and CARLA to validate the
effectiveness of our method. We show that our method significantly improves
generalization capability while only marginally affecting sample efficiency. In
particular, when integrated with DrQ-v2, our method enhances the test
performance of DrQ-v2 on CARLA across various scenarios, from 14% of the
training performance to 97%.
Related papers
- Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder [12.310140622800372]
We propose APE: efficient reinforcement learning through Adaptively Pretrained visual.
APE uses adaptive augmentation strategy during the pretraining phase and extracts generalizable features with only a few interactions within the task environments in the policy learning period.
Results show that mainstream RL methods, such as DreamerV3 and DrQ-v2, achieve state-of-the-art performance when equipped with APE.
arXiv Detail & Related papers (2025-02-08T12:57:02Z) - SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training [127.47044960572659]
Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models.
This paper studies the difference between SFT and RL on generalization and memorization.
We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants.
arXiv Detail & Related papers (2025-01-28T18:59:44Z) - RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning [53.8293458872774]
We propose Reinforcement Learning Distilled Generalists (RLDG) to generate high-quality training data for finetuning generalist policies.
We demonstrate that generalist policies trained with RL-generated data consistently outperform those trained with human demonstrations.
Our results suggest that combining task-specific RL with generalist policy distillation offers a promising approach for developing more capable and efficient robotic manipulation systems.
arXiv Detail & Related papers (2024-12-13T04:57:55Z) - IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning [17.236861687708096]
Continual learning (CL) remains one of the long-standing challenges for deep neural networks due to catastrophic forgetting of previously acquired knowledge.
Inspired by how humans learn using strong inductive biases, we propose IMEX-Reg to improve the generalization performance of experience rehearsal in CL under low buffer regimes.
arXiv Detail & Related papers (2024-04-28T12:25:09Z) - Sample Efficient Myopic Exploration Through Multitask Reinforcement
Learning with Diverse Tasks [53.44714413181162]
This paper shows that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design can be sample-efficient.
To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL.
arXiv Detail & Related papers (2024-03-03T22:57:44Z) - Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms.
We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
arXiv Detail & Related papers (2023-04-20T17:11:05Z) - Local Feature Swapping for Generalization in Reinforcement Learning [0.0]
We introduce a new regularization technique consisting of channel-consistent local permutations (CLOP) of the feature maps.
The proposed permutations induce robustness to spatial correlations and help prevent overfitting behaviors in reinforcement learning.
We demonstrate, on the OpenAI Procgen Benchmark, that RL agents trained with the CLOP method exhibit robustness to visual changes and better generalization properties.
arXiv Detail & Related papers (2022-04-13T13:12:51Z) - Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit
Partial Observability [92.95794652625496]
Generalization is a central challenge for the deployment of reinforcement learning systems.
We show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial observability.
We recast the problem of generalization in RL as solving the induced partially observed Markov decision process.
arXiv Detail & Related papers (2021-07-13T17:59:25Z) - Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under
Data Augmentation [25.493902939111265]
We investigate causes of instability when using data augmentation in off-policy Reinforcement Learning algorithms.
We propose a simple yet effective technique for stabilizing this class of algorithms under augmentation.
Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL.
arXiv Detail & Related papers (2021-07-01T17:58:05Z) - How to Make Deep RL Work in Practice [15.740760669623876]
Reported results of state-of-the-art algorithms are often difficult to reproduce.
We make suggestions which of those techniques to use by default and highlight areas that could benefit from a solution specifically tailored to RL.
arXiv Detail & Related papers (2020-10-25T10:37:54Z) - Dynamics Generalization via Information Bottleneck in Deep Reinforcement
Learning [90.93035276307239]
We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents.
We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks.
This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
arXiv Detail & Related papers (2020-08-03T02:24:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.