Multi-Task Interactive Robot Fleet Learning with Visual World Models

• URL: http://arxiv.org/abs/2410.22689v1
• Date: Wed, 30 Oct 2024 04:49:39 GMT
• Title: Multi-Task Interactive Robot Fleet Learning with Visual World Models
• Authors: Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, Yuke Zhu,
• Abstract summary: Sirius-Fleet is a multi-task interactive robot fleet learning framework. It monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. As the robot autonomy improves, anomaly predictors automatically adapt their prediction criteria.
• Score: 25.001148860168477
• License:
• Abstract: Recent advancements in large-scale multi-task robot learning offer the potential for deploying robot fleets in household and industrial settings, enabling them to perform diverse tasks across various environments. However, AI-enabled robots often face challenges with generalization and robustness when exposed to real-world variability and uncertainty. We introduce Sirius-Fleet, a multi-task interactive robot fleet learning framework to address these challenges. Sirius-Fleet monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. We employ a visual world model to predict the outcomes of future actions and build anomaly predictors to predict whether they will likely result in anomalies. As the robot autonomy improves, the anomaly predictors automatically adapt their prediction criteria, leading to fewer requests for human intervention and gradually reducing human workload over time. Evaluations on large-scale benchmarks demonstrate Sirius-Fleet's effectiveness in improving multi-task policy performance and monitoring accuracy. We demonstrate Sirius-Fleet's performance in both RoboCasa in simulation and Mutex in the real world, two diverse, large-scale multi-task benchmarks. More information is available on the project website: https://ut-austin-rpl.github.io/sirius-fleet

Related papers

• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Robotic Control via Embodied Chain-of-Thought Reasoning [86.6680905262442]
  Key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models can substantially improve their robustness and generalization ability. We introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features before predicting the robot action.
  arXiv  Detail & Related papers  (2024-07-11T17:31:01Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• Innate Motivation for Robot Swarms by Minimizing Surprise: From Simple Simulations to Real-World Experiments [6.21540494241516]
  Large-scale mobile multi-robot systems can be beneficial over monolithic robots because of higher potential for robustness and scalability. Developing controllers for multi-robot systems is challenging because the multitude of interactions is hard to anticipate and difficult to model. Innate motivation tries to avoid the specific formulation of rewards and work instead with different drivers, such as curiosity. A unique advantage of the swarm robot case is that swarm members populate the robot's environment and can trigger more active behaviors in a self-referential loop.
  arXiv  Detail & Related papers  (2024-05-04T06:25:58Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment [25.186525630548356]
  Sirius is a principled framework for humans and robots to collaborate through a division of work. Partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably. We introduce a new learning algorithm to improve the policy's performance on the data collected from the task executions.
  arXiv  Detail & Related papers  (2022-11-15T18:53:39Z)
• Robot Vitals and Robot Health: Towards Systematically Quantifying Runtime Performance Degradation in Robots Under Adverse Conditions [2.0625936401496237]
  "Robot vitals" are indicators that estimate the extent of performance degradation faced by a robot. "Robot health" is a metric that combines robot vitals into a single scalar value estimate of performance degradation.
  arXiv  Detail & Related papers  (2022-07-04T19:26:13Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.