Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning

• URL: http://arxiv.org/abs/2005.12254v1
• Date: Mon, 25 May 2020 17:56:08 GMT
• Title: Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Authors: Jaskirat Singh and Liang Zheng
• Abstract summary: Training deep reinforcement learning agents on environments with multiple levels / scenes / conditions from the same task, has become essential for many applications. We propose a dynamic value estimation (DVE) technique for these multiple-MDP environments, motivated by the clustering effect observed in the value function distribution across different scenes.
• Score: 22.889059874754242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training deep reinforcement learning agents on environments with multiple levels / scenes / conditions from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods continue to view this collection of scenes as a single Markov Decision Process (MDP) with a common value function; however, we argue that it is better to treat the collection as a single environment with multiple underlying MDPs. To this end, we propose a dynamic value estimation (DVE) technique for these multiple-MDP environments, motivated by the clustering effect observed in the value function distribution across different scenes. The resulting agent is able to learn a more accurate and scene-specific value function estimate (and hence the advantage function), leading to a lower sample variance. Our proposed approach is simple to accommodate with several existing implementations (like PPO, A3C) and results in consistent improvements for a range of ProcGen environments and the AI2-THOR framework based visual navigation task.

Related papers

• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification [26.896926631411652]
  Multi-Instance Learning (MIL) has shown impressive performance for histopathology whole slide image (WSI) analysis using bags or pseudo-bags. Existing MIL-based technologies at least suffer from one or more of the following problems: 1) requiring high storage and intensive pre-processing for numerous instances (sampling); 2) potential over-fitting with limited knowledge to predict bag labels (feature representation); 3) pseudo-bag counts and prior biases affect model robustness and generalizability (decision-making)
  arXiv  Detail & Related papers  (2024-03-09T04:43:24Z)
• Multi-Level Contrastive Learning for Dense Prediction Task [59.591755258395594]
  We present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. Our method consistently outperforms the recent state-of-the-art methods on various datasets with significant margins.
  arXiv  Detail & Related papers  (2023-04-04T17:59:04Z)
• Addressing the issue of stochastic environments and local decision-making in multi-objective reinforcement learning [0.0]
  Multi-objective reinforcement learning (MORL) is a relatively new field which builds on conventional Reinforcement Learning (RL) This thesis focuses on what factors influence the frequency with which value-based MORL Q-learning algorithms learn the optimal policy for an environment.
  arXiv  Detail & Related papers  (2022-11-16T04:56:42Z)
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning [16.910911657616005]
  Training deep reinforcement learning agents on environments with multiple levels / scenes from the same task, has become essential for many applications. We argue that the sample variance for a multi-scene environment is best minimized by treating each scene as a distinct MDP. We also demonstrate that the true joint value function for a multi-scene environment, follows a multi-modal distribution which is not captured by traditional CNN / LSTM based critic networks.
  arXiv  Detail & Related papers  (2021-02-14T23:30:13Z)
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation [22.889059874754242]
  Multi-scene reinforcement learning has become essential for many applications. One strategy for variance reduction is to consider each scene as a distinct Markov decision process (MDP) In this paper, we argue that the error between the true scene-specific value function and the predicted dynamic estimate can be further reduced by progressively enforcing sparse cluster assignments.
  arXiv  Detail & Related papers  (2020-11-25T08:35:16Z)
• Multi-Scale Positive Sample Refinement for Few-Shot Object Detection [61.60255654558682]
  Few-shot object detection (FSOD) helps detectors adapt to unseen classes with few training instances. We propose a Multi-scale Positive Sample Refinement (MPSR) approach to enrich object scales in FSOD. MPSR generates multi-scale positive samples as object pyramids and refines the prediction at various scales.
  arXiv  Detail & Related papers  (2020-07-18T09:48:29Z)
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs [55.31018404591743]
  We leverage ideas of common structure from the HiP-MDP setting to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings.
  arXiv  Detail & Related papers  (2020-07-14T17:25:27Z)
• Unsupervised Domain Adaptation in Person re-ID via k-Reciprocal Clustering and Large-Scale Heterogeneous Environment Synthesis [76.46004354572956]
  We introduce an unsupervised domain adaptation approach for person re-identification. Experimental results show that the proposed ktCUDA and SHRED approach achieves an average improvement of +5.7 mAP in re-identification performance.
  arXiv  Detail & Related papers  (2020-01-14T17:43:52Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.