Adventurer: Exploration with BiGAN for Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2503.18612v1
• Date: Mon, 24 Mar 2025 12:13:24 GMT
• Title: Adventurer: Exploration with BiGAN for Deep Reinforcement Learning
• Authors: Yongshuai Liu, Xin Liu,
• Abstract summary: We show that BiGAN performs well in estimating state novelty for complex observations.<n>Our results show that Adventurer produces competitive results on a range of popular benchmark tasks.
• Score: 4.902161835372679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent developments in deep reinforcement learning have been very successful in learning complex, previously intractable problems. Sample efficiency and local optimality, however, remain significant challenges. To address these challenges, novelty-driven exploration strategies have emerged and shown promising potential. Unfortunately, no single algorithm outperforms all others in all tasks and most of them struggle with tasks with high-dimensional and complex observations. In this work, we propose Adventurer, a novelty-driven exploration algorithm that is based on Bidirectional Generative Adversarial Networks (BiGAN), where BiGAN is trained to estimate state novelty. Intuitively, a generator that has been trained on the distribution of visited states should only be able to generate a state coming from the distribution of visited states. As a result, novel states using the generator to reconstruct input states from certain latent representations would lead to larger reconstruction errors. We show that BiGAN performs well in estimating state novelty for complex observations. This novelty estimation method can be combined with intrinsic-reward-based exploration. Our empirical results show that Adventurer produces competitive results on a range of popular benchmark tasks, including continuous robotic manipulation tasks (e.g. Mujoco robotics) and high-dimensional image-based tasks (e.g. Atari games).

Related papers

• Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models [5.404186221463082]
  Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems.<n>We propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore.<n>IGE has a human-like ability to instinctively identify how interesting or promising any new state is.
  arXiv  Detail & Related papers  (2024-05-24T01:45:27Z)
• Sample Efficient Deep Reinforcement Learning via Local Planning [21.420851589712626]
  This work focuses on sample-efficient deep reinforcement learning (RL) with a simulator. We propose an algorithmic framework, named uncertainty-first local planning (UFLP), that takes advantage of this property. We demonstrate that this simple procedure can dramatically improve the sample cost of several baseline RL algorithms on difficult exploration tasks.
  arXiv  Detail & Related papers  (2023-01-29T23:17:26Z)
• Generative Adversarial Exploration for Reinforcement Learning [48.379457575356454]
  In this paper, we propose a novel method called generative adversarial exploration (GAEX) to encourage exploration in reinforcement learning (RL) In our experiments, we apply GAEX to the game Venture, Montezuma's Revenge and Super Mario Bros. To our knowledge, this is the first work to employ GAN in RL exploration problems.
  arXiv  Detail & Related papers  (2022-01-27T17:34:47Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)
• Self-Supervised Exploration via Latent Bayesian Surprise [4.088019409160893]
  In this work, we propose a curiosity-based bonus as intrinsic reward for Reinforcement Learning. We extensively evaluate our model by measuring the agent's performance in terms of environment exploration. Our model is cheap and empirically shows state-of-the-art performance on several problems.
  arXiv  Detail & Related papers  (2021-04-15T14:40:16Z)
• Learning to Shift Attention for Motion Generation [55.61994201686024]
  One challenge of motion generation using robot learning from demonstration techniques is that human demonstrations follow a distribution with multiple modes for one task query. Previous approaches fail to capture all modes or tend to average modes of the demonstrations and thus generate invalid trajectories. We propose a motion generation model with extrapolation ability to overcome this problem.
  arXiv  Detail & Related papers  (2021-02-24T09:07:52Z)
• Batch Exploration with Examples for Scalable Robotic Reinforcement Learning [63.552788688544254]
  Batch Exploration with Examples (BEE) explores relevant regions of the state-space guided by a modest number of human provided images of important states. BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot.
  arXiv  Detail & Related papers  (2020-10-22T17:49:25Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• First return, then explore [18.876005532689234]
  Go-Explore is a family of algorithms that explicitly remembers promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games. We show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handleity throughout training.
  arXiv  Detail & Related papers  (2020-04-27T16:31:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.