A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points

• URL: http://arxiv.org/abs/2405.14214v1
• Date: Thu, 23 May 2024 06:17:26 GMT
• Title: A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points
• Authors: Zihe Liu, Jie Lu, Guangquan Zhang, Junyu Xuan,
• Abstract summary: This research introduces Behavior-Aware Detection and Adaptation (BADA), an innovative framework that merges environmental change detection with behavior adaptation. The key inspiration behind our method is that policies exhibit different global behaviors in changing environments. The results of a series of experiments demonstrate better performance relative to several current algorithms.
• Score: 30.077746056549678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep reinforcement learning is used in various domains, but usually under the assumption that the environment has stationary conditions like transitions and state distributions. When this assumption is not met, performance suffers. For this reason, tracking continuous environmental changes and adapting to unpredictable conditions is challenging yet crucial because it ensures that systems remain reliable and flexible in practical scenarios. Our research introduces Behavior-Aware Detection and Adaptation (BADA), an innovative framework that merges environmental change detection with behavior adaptation. The key inspiration behind our method is that policies exhibit different global behaviors in changing environments. Specifically, environmental changes are identified by analyzing variations between behaviors using Wasserstein distances without manually set thresholds. The model adapts to the new environment through behavior regularization based on the extent of changes. The results of a series of experiments demonstrate better performance relative to several current algorithms. This research also indicates significant potential for tackling this long-standing challenge.

Related papers

• Continuously evolving rewards in an open-ended environment [0.0]
  RULE: Reward Updating through Learning and Expectation is tested in a simplified ecosystem-like setting. The population of entities successfully demonstrate the abandonment of an initially rewarded but ultimately detrimental behaviour. These adjustment happen through endogenous modification of the entities' underlying reward function, during continuous learning, without external intervention.
  arXiv  Detail & Related papers  (2024-05-02T13:07:56Z)
• HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments [93.94020724735199]
  HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines.
  arXiv  Detail & Related papers  (2024-01-23T18:59:43Z)
• Invariant Causal Imitation Learning for Generalizable Policies [87.51882102248395]
  We propose Invariant Causal Learning (ICIL) to learn an imitation policy. ICIL learns a representation of causal features that is disentangled from the specific representations of noise variables. We show that ICIL is effective in learning imitation policies capable of generalizing to unseen environments.
  arXiv  Detail & Related papers  (2023-11-02T16:52:36Z)
• AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning [13.167123175701802]
  This paper formalizes the task of adapting to changing environmental dynamics in Reinforcement Learning (RL) We then propose the Asymmetric Actor-Critic in Contextual RL (AACC) as an end-to-end actor-critic method to deal with such generalization tasks. We demonstrate the essential improvements in the performance of AACC over existing baselines experimentally in a range of simulated environments.
  arXiv  Detail & Related papers  (2022-08-03T22:52:26Z)
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL [142.36621929739707]
  We show that learning diverse behaviors for accomplishing a task can lead to behavior that generalizes to varying environments. By identifying multiple solutions for the task in a single environment during training, our approach can generalize to new situations.
  arXiv  Detail & Related papers  (2020-10-27T17:41:57Z)
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning [12.76337275628074]
  In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality andgenerativeity. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration. Our method outperforms several state-of-the-art environment model-based exploration approaches.
  arXiv  Detail & Related papers  (2020-10-17T09:54:51Z)
• Self-Supervised Policy Adaptation during Deployment [98.25486842109936]
  Self-supervision allows the policy to continue training after deployment without using any rewards. Empirical evaluations are performed on diverse simulation environments from DeepMind Control suite and ViZDoom. Our method improves generalization in 31 out of 36 environments across various tasks and outperforms domain randomization on a majority of environments.
  arXiv  Detail & Related papers  (2020-07-08T17:56:27Z)
• Ecological Reinforcement Learning [76.9893572776141]
  We study the kinds of environment properties that can make learning under such conditions easier. understanding how properties of the environment impact the performance of reinforcement learning agents can help us to structure our tasks in ways that make learning tractable.
  arXiv  Detail & Related papers  (2020-06-22T17:55:03Z)
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity [67.24635298387624]
  We show that an off-policy RL algorithm can reason about and tackle lifelong non-stationarity. Our method leverages latent variable models to learn a representation of the environment from current and past experiences. We also introduce several simulation environments that exhibit lifelong non-stationarity, and empirically find that our approach substantially outperforms approaches that do not reason about environment shift.
  arXiv  Detail & Related papers  (2020-06-18T17:34:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.