Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks

• URL: http://arxiv.org/abs/2410.23208v1
• Date: Wed, 30 Oct 2024 16:59:41 GMT
• Title: Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
• Authors: Michael Matthews, Michael Beukman, Chris Lu, Jakob Foerster,
• Abstract summary: We procedurally generate tens of millions of 2D physics-based tasks and use these to train a general reinforcement learning (RL) agent for physical control. Kinetix is an open-ended space of physics-based RL environments that can represent tasks ranging from robotic locomotion and grasping to video games and classic RL environments. Our trained agent exhibits strong physical reasoning capabilities, being able to zero-shot solve unseen human-designed environments.
• Score: 3.479490713357225
• License:
• Abstract: While large models trained with self-supervised learning on offline datasets have shown remarkable capabilities in text and image domains, achieving the same generalisation for agents that act in sequential decision problems remains an open challenge. In this work, we take a step towards this goal by procedurally generating tens of millions of 2D physics-based tasks and using these to train a general reinforcement learning (RL) agent for physical control. To this end, we introduce Kinetix: an open-ended space of physics-based RL environments that can represent tasks ranging from robotic locomotion and grasping to video games and classic RL environments, all within a unified framework. Kinetix makes use of our novel hardware-accelerated physics engine Jax2D that allows us to cheaply simulate billions of environment steps during training. Our trained agent exhibits strong physical reasoning capabilities, being able to zero-shot solve unseen human-designed environments. Furthermore, fine-tuning this general agent on tasks of interest shows significantly stronger performance than training an RL agent *tabula rasa*. This includes solving some environments that standard RL training completely fails at. We believe this demonstrates the feasibility of large scale, mixed-quality pre-training for online RL and we hope that Kinetix will serve as a useful framework to investigate this further.

Related papers

• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL. On real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 20K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
  This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents. We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
  arXiv  Detail & Related papers  (2024-06-14T17:49:55Z)
• Active Reinforcement Learning for Robust Building Control [0.0]
  Reinforcement learning (RL) is a powerful tool for optimal control that has found great success in Atari games, the game of Go, robotic control, and building optimization. Unsupervised environment design (UED) has been proposed as a solution to this problem, in which the agent trains in environments that have been specially selected to help it learn. We show that ActivePLR is able to outperform state-of-the-art UED algorithms in minimizing energy usage while maximizing occupant comfort in the setting of building control.
  arXiv  Detail & Related papers  (2023-12-16T02:18:45Z)
• Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning [66.10854214036605]
  A central question in robotics is how to design a control system for an agile mobile robot. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour.
  arXiv  Detail & Related papers  (2023-10-17T02:40:27Z)
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data [101.43350024175157]
  Self-supervised learning has the potential to decrease the amount of human annotation and engineering effort required to learn control strategies. Our work builds on prior work showing that the reinforcement learning (RL) itself can be cast as a self-supervised problem. We demonstrate that a self-supervised RL algorithm based on contrastive learning can solve real-world, image-based robotic manipulation tasks.
  arXiv  Detail & Related papers  (2023-06-06T01:36:56Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation [4.597465975849579]
  Reinforcement learning (RL) is an agent-based approach for teaching robots to navigate within the physical world. We present a simulation framework, built on AirSim, which provides efficient parallel training. Building on this framework, Ape-X is modified to incorporate decentralised training of AirSim environments.
  arXiv  Detail & Related papers  (2022-09-22T15:27:42Z)
• CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems [3.2550963598419957]
  CyGIL is an experimental testbed of an emulated RL training environment for network cyber operations. It uses a stateless environment architecture and incorporates the MITRE ATT&CK framework to establish a high fidelity training environment. Its comprehensive action space and flexible game design allow the agent training to focus on particular advanced persistent threat (APT) profiles.
  arXiv  Detail & Related papers  (2021-09-07T20:52:44Z)
• Robust Reinforcement Learning-based Autonomous Driving Agent for Simulation and Real World [0.0]
  We present a DRL-based algorithm that is capable of performing autonomous robot control using Deep Q-Networks (DQN) In our approach, the agent is trained in a simulated environment and it is able to navigate both in a simulated and real-world environment. The trained agent is able to run on limited hardware resources and its performance is comparable to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-09-23T15:23:54Z)
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real [74.45688231140689]
  We introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. We obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T08:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.