Effective Diversity in Population Based Reinforcement Learning

• URL: http://arxiv.org/abs/2002.00632v3
• Date: Wed, 7 Oct 2020 16:03:39 GMT
• Title: Effective Diversity in Population Based Reinforcement Learning
• Authors: Jack Parker-Holder and Aldo Pacchiano and Krzysztof Choromanski and Stephen Roberts
• Abstract summary: We introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold. Our algorithm Diversity via Determinants (DvD) adapts the degree of diversity during training using online learning techniques.
• Score: 38.62641968788987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.

Related papers

• DIDI: Diffusion-Guided Diversity for Offline Behavioral Generation [25.904918670006587]
  We propose a novel approach called DIffusion-guided DIversity (DIDI) for offline behavioral generation. The goal of DIDI is to learn a diverse set of skills from a mixture of label-free offline data.
  arXiv  Detail & Related papers  (2024-05-23T17:00:15Z)
• Improving Generalization of Alignment with Human Preferences through Group Invariant Learning [56.19242260613749]
  Reinforcement Learning from Human Feedback (RLHF) enables the generation of responses more aligned with human preferences. Previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. We propose a novel approach that can learn a consistent policy via RL across various data groups or domains.
  arXiv  Detail & Related papers  (2023-10-18T13:54:15Z)
• Diversity from Human Feedback [39.05609489642456]
  We propose the problem of learning a behavior space from human feedback and present a general method called Diversity from Human Feedback (DivHF) to solve it. DivHF learns a behavior consistent with human preference by querying human feedback. We demonstrate the effectiveness of DivHF by integrating it with the Quality-Diversity optimization algorithm MAP-Elites and conducting experiments on the QDax suite.
  arXiv  Detail & Related papers  (2023-10-10T14:13:59Z)
• Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning [24.36351102003414]
  Human activity recognition (HAR) is a time series classification task that focuses on identifying the motion patterns from human sensor readings. We propose a novel approach called Diverse and Discriminative representation Learning (DDLearn) for generalizable lowresource HAR. Our method significantly outperforms state-of-art methods by an average accuracy improvement of 9.5%.
  arXiv  Detail & Related papers  (2023-05-25T08:24:22Z)
• System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning [8.280943341629161]
  We introduce System Neural Diversity (SND): a measure of behavioral heterogeneity in multi-agent systems. We show how SND allows us to measure latent resilience skills acquired by the agents, while other proxies, such as task performance (reward), fail. We demonstrate how this paradigm can be used to bootstrap the exploration phase, finding optimal policies faster.
  arXiv  Detail & Related papers  (2023-05-03T13:58:13Z)
• Source-free Domain Adaptation Requires Penalized Diversity [60.04618512479438]
  Source-free domain adaptation (SFDA) was introduced to address knowledge transfer between different domains in the absence of source data. In unsupervised SFDA, the diversity is limited to learning a single hypothesis on the source or learning multiple hypotheses with a shared feature extractor. We propose a novel unsupervised SFDA algorithm that promotes representational diversity through the use of separate feature extractors.
  arXiv  Detail & Related papers  (2023-04-06T00:20:19Z)
• Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition [63.67574523750839]
  We propose a generic reinforcement learning (RL) algorithm that performs better than baseline deep Q-learning algorithms in environments with multiple variably-valued niches. We show that agents trained this way can escape poor-but-attractive local optima to instead converge to harder-to-discover higher value strategies.
  arXiv  Detail & Related papers  (2023-02-02T16:00:19Z)
• Multi-Domain Joint Training for Person Re-Identification [51.73921349603597]
  Deep learning-based person Re-IDentification (ReID) often requires a large amount of training data to achieve good performance. It appears that collecting more training data from diverse environments tends to improve the ReID performance. We propose an approach called Domain-Camera-Sample Dynamic network (DCSD) whose parameters can be adaptive to various factors.
  arXiv  Detail & Related papers  (2022-01-06T09:20:59Z)
• Quantifying environment and population diversity in multi-agent reinforcement learning [7.548322030720646]
  Generalization is a major challenge for multi-agent reinforcement learning. In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity.
  arXiv  Detail & Related papers  (2021-02-16T18:54:39Z)
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation [109.11580756757611]
  Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members' performances. We introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features.
  arXiv  Detail & Related papers  (2021-01-14T10:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.