Improving Generalization in Reinforcement Learning with Mixture Regularization

• URL: http://arxiv.org/abs/2010.10814v1
• Date: Wed, 21 Oct 2020 08:12:03 GMT
• Title: Improving Generalization in Reinforcement Learning with Mixture Regularization
• Abstract summary: We introduce a simple approach, named mixreg, which trains agents on a mixture of observations from different training environments. Mixreg increases the data diversity more effectively and helps learn smoother policies. Results show mixreg outperforms the well-established baselines on unseen testing environments by a large margin.
• Score: 113.12412071717078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep reinforcement learning (RL) agents trained in a limited set of environments tend to suffer overfitting and fail to generalize to unseen testing environments. To improve their generalizability, data augmentation approaches (e.g. cutout and random convolution) are previously explored to increase the data diversity. However, we find these approaches only locally perturb the observations regardless of the training environments, showing limited effectiveness on enhancing the data diversity and the generalization performance. In this work, we introduce a simple approach, named mixreg, which trains agents on a mixture of observations from different training environments and imposes linearity constraints on the observation interpolations and the supervision (e.g. associated reward) interpolations. Mixreg increases the data diversity more effectively and helps learn smoother policies. We verify its effectiveness on improving generalization by conducting extensive experiments on the large-scale Procgen benchmark. Results show mixreg outperforms the well-established baselines on unseen testing environments by a large margin. Mixreg is simple, effective and general. It can be applied to both policy-based and value-based RL algorithms. Code is available at https://github.com/kaixin96/mixreg .

Related papers

• Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning [3.279378558656342]
  Reinforcement learning (RL) often suffers from performance degradation when deployed in environments that differ from those encountered during training.<n>Existing techniques such as domain randomization (DR) mitigate this, but require access to diverse training environments and full trajectory observability.<n>We propose Generalization via Evolutionary Reward Shaping (GERS), a bilevel optimization approach to improve generalization on unseen test environments.
  arXiv  Detail & Related papers  (2026-06-15T05:35:16Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity [4.98310859187348]
  Federated reinforcement learning (FedRL) enables multiple agents to collaboratively train a global policy without sharing raw data.<n>FedRL faces challenges in heterogeneous environments where differing state-transition dynamics lead to non-identical input distributions.<n>This paper develops a personalized observation normalization (PON) method, allowing each agent to locally normalize raw state inputs.
  arXiv  Detail & Related papers  (2026-04-10T19:37:16Z)
• Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression [18.522115769904477]
  We propose a principled integration of multi-task learning using multi-output Gaussian processes (MOGP) at the local level and federated learning at the global level. Challenges in performing posterior inference on local devices are addressed through the P'o'lya-Gamma augmentation technique and mean-field variational inference. Experimental results on both synthetic and real data demonstrate superior predictive performance, OOD detection, uncertainty calibration and convergence rate.
  arXiv  Detail & Related papers  (2024-12-14T17:10:54Z)
• Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning [10.00979536266327]
  We present a novel approach to address the challenge of generalization in offline reinforcement learning (RL) Specifically, we aim to improve the agent's ability to generalize to out-of-distribution goals. We learn a new policy offline based on the augmented dataset, with an off-the-shelf offline RL algorithm.
  arXiv  Detail & Related papers  (2023-09-14T10:22:33Z)
• Expeditious Saliency-guided Mix-up through Random Gradient Thresholding [89.59134648542042]
  Mix-up training approaches have proven to be effective in improving the generalization ability of Deep Neural Networks. In this paper, inspired by the superior qualities of each direction over one another, we introduce a novel method that lies at the junction of the two routes. We name our method R-Mix following the concept of "Random Mix-up" In order to address the question of whether there exists a better decision protocol, we train a Reinforcement Learning agent that decides the mix-up policies.
  arXiv  Detail & Related papers  (2022-12-09T14:29:57Z)
• FIXED: Frustratingly Easy Domain Generalization with Mixup [53.782029033068675]
  Domain generalization (DG) aims to learn a generalizable model from multiple training domains such that it can perform well on unseen target domains. A popular strategy is to augment training data to benefit generalization through methods such as Mixupcitezhang 2018mixup. We propose a simple yet effective enhancement for Mixup-based DG, namely domain-invariant Feature mIXup (FIX) Our approach significantly outperforms nine state-of-the-art related methods, beating the best performing baseline by 6.5% on average in terms of test accuracy.
  arXiv  Detail & Related papers  (2022-11-07T09:38:34Z)
• C-Mixup: Improving Generalization in Regression [71.10418219781575]
  Mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. We propose C-Mixup, which adjusts the sampling probability based on the similarity of the labels. C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively.
  arXiv  Detail & Related papers  (2022-10-11T20:39:38Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• Implicit Gradient Alignment in Distributed and Federated Learning [39.61762498388211]
  A major obstacle to achieving global convergence in distributed and federated learning is misalignment of gradients across clients. We propose a novel GradAlign algorithm that induces the same implicit regularization while allowing the use of arbitrarily large batches in each update.
  arXiv  Detail & Related papers  (2021-06-25T22:01:35Z)
• MixRL: Data Mixing Augmentation for Regression using Reinforcement Learning [2.1345682889327837]
  Existing techniques for data augmentation largely focus on classification tasks and do not readily apply to regression tasks. We show that mixing examples that either have a large data or label distance may have an increasingly-negative effect on model performance. We propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors it should be mixed with for the best model performance.
  arXiv  Detail & Related papers  (2021-06-07T07:01:39Z)
• k-Mixup Regularization for Deep Learning via Optimal Transport [32.951696405505686]
  Mixup is a popular regularization technique for training deep neural networks. We extend mixup in a simple, broadly applicable way to emph$k$-mixup, which perturbs $k$-batches of training points in the direction of other $k$-batches. We show that training with $k$-mixup further improves generalization and robustness across several network architectures.
  arXiv  Detail & Related papers  (2021-06-05T17:08:08Z)
• Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity [15.780905917870427]
  We propose a new perspective on batch mixup and formulate the optimal construction of a batch of mixup data. We also propose an efficient modular approximation based iterative submodular computation algorithm for efficient mixup per each minibatch. Our experiments show the proposed method achieves the state of the art generalization, calibration, and weakly supervised localization results.
  arXiv  Detail & Related papers  (2021-02-05T09:12:02Z)
• Unshuffling Data for Improved Generalization [65.57124325257409]
  Generalization beyond the training distribution is a core challenge in machine learning. We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
  arXiv  Detail & Related papers  (2020-02-27T03:07:41Z)
• Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy [77.34280933613226]
  We propose a general regularizer called textbfPatch-level Neighborhood Interpolation(Pani) that conducts a non-local representation in the computation of networks. Our proposal explicitly constructs patch-level graphs in different layers and then linearly interpolates neighborhood patch features, serving as a general and effective regularization strategy.
  arXiv  Detail & Related papers  (2019-11-21T06:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.