Revisiting Data Augmentation in Deep Reinforcement Learning
- URL: http://arxiv.org/abs/2402.12181v1
- Date: Mon, 19 Feb 2024 14:42:10 GMT
- Title: Revisiting Data Augmentation in Deep Reinforcement Learning
- Authors: Jianshu Hu, Yunpeng Jiang and Paul Weng
- Abstract summary: Various data augmentation techniques have been recently proposed in image-based deep reinforcement learning (DRL)
We analyze existing methods to better understand them and to uncover how they are connected.
This analysis suggests recommendations on how to exploit data augmentation in a more principled way.
- Score: 3.660182910533372
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Various data augmentation techniques have been recently proposed in
image-based deep reinforcement learning (DRL). Although they empirically
demonstrate the effectiveness of data augmentation for improving sample
efficiency or generalization, which technique should be preferred is not always
clear. To tackle this question, we analyze existing methods to better
understand them and to uncover how they are connected. Notably, by expressing
the variance of the Q-targets and that of the empirical actor/critic losses of
these methods, we can analyze the effects of their different components and
compare them. We furthermore formulate an explanation about how these methods
may be affected by choosing different data augmentation transformations in
calculating the target Q-values. This analysis suggests recommendations on how
to exploit data augmentation in a more principled way. In addition, we include
a regularization term called tangent prop, previously proposed in computer
vision, but whose adaptation to DRL is novel to the best of our knowledge. We
evaluate our proposition and validate our analysis in several domains. Compared
to different relevant baselines, we demonstrate that it achieves
state-of-the-art performance in most environments and shows higher sample
efficiency and better generalization ability in some complex environments.
Related papers
- See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition [56.87609859444084]
parameter-efficient fine-tuning (PEFT) focuses on optimizing a select subset of parameters while keeping the rest fixed, significantly lowering computational and storage overheads.
We take the first step to unify all approaches by dissecting them from a decomposition perspective.
We introduce two novel PEFT methods alongside a simple yet effective framework designed to enhance the performance of PEFT techniques across various applications.
arXiv Detail & Related papers (2024-07-07T15:44:42Z) - On Counterfactual Data Augmentation Under Confounding [30.76982059341284]
Counterfactual data augmentation has emerged as a method to mitigate confounding biases in the training data.
These biases arise due to various observed and unobserved confounding variables in the data generation process.
We show how our simple augmentation method helps existing state-of-the-art methods achieve good results.
arXiv Detail & Related papers (2023-05-29T16:20:23Z) - Augmentation Invariant Manifold Learning [0.5827521884806071]
We introduce a new representation learning method called augmentation invariant manifold learning.
Compared with existing self-supervised methods, the new method simultaneously exploits the manifold's geometric structure and invariant property of augmented data.
Our theoretical investigation characterizes the role of data augmentation in the proposed method and reveals why and how the data representation learned from augmented data can improve the $k$-nearest neighbor in the downstream analysis.
arXiv Detail & Related papers (2022-11-01T13:42:44Z) - Automatic Data Augmentation via Invariance-Constrained Learning [94.27081585149836]
Underlying data structures are often exploited to improve the solution of learning tasks.
Data augmentation induces these symmetries during training by applying multiple transformations to the input data.
This work tackles these issues by automatically adapting the data augmentation while solving the learning task.
arXiv Detail & Related papers (2022-09-29T18:11:01Z) - CCLF: A Contrastive-Curiosity-Driven Learning Framework for
Sample-Efficient Reinforcement Learning [56.20123080771364]
We develop a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF) for reinforcement learning.
CCLF fully exploit sample importance and improve learning efficiency in a self-supervised manner.
We evaluate this approach on the DeepMind Control Suite, Atari, and MiniGrid benchmarks.
arXiv Detail & Related papers (2022-05-02T14:42:05Z) - Reenvisioning Collaborative Filtering vs Matrix Factorization [65.74881520196762]
Collaborative filtering models based on matrix factorization and learned similarities using Artificial Neural Networks (ANNs) have gained significant attention in recent years.
Announcement of ANNs within the recommendation ecosystem has been recently questioned, raising several comparisons in terms of efficiency and effectiveness.
We show the potential these techniques may have on beyond-accuracy evaluation while analyzing effect on complementary evaluation dimensions.
arXiv Detail & Related papers (2021-07-28T16:29:38Z) - Removing Undesirable Feature Contributions Using Out-of-Distribution
Data [20.437871747430826]
We propose a data augmentation method to improve generalization in both adversarial and standard learning.
The proposed method can further improve the existing state-of-the-art adversarial training.
arXiv Detail & Related papers (2021-01-17T10:26:34Z) - Generalization in Reinforcement Learning by Soft Data Augmentation [11.752595047069505]
SOft Data Augmentation (SODA) is a method that decouples augmentation from policy learning.
We find SODA to significantly advance sample efficiency, generalization, and stability in training over state-of-the-art vision-based RL methods.
arXiv Detail & Related papers (2020-11-26T17:00:34Z) - On the Benefits of Invariance in Neural Networks [56.362579457990094]
We show that training with data augmentation leads to better estimates of risk and thereof gradients, and we provide a PAC-Bayes generalization bound for models trained with data augmentation.
We also show that compared to data augmentation, feature averaging reduces generalization error when used with convex losses, and tightens PAC-Bayes bounds.
arXiv Detail & Related papers (2020-05-01T02:08:58Z) - Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting.
G-DAUGC consistently outperforms existing data augmentation methods based on back-translation.
Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
arXiv Detail & Related papers (2020-04-24T06:12:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.