Never-Ending Behavior-Cloning Agent for Robotic Manipulation

• URL: http://arxiv.org/abs/2403.00336v2
• Date: Fri, 7 Jun 2024 08:10:11 GMT
• Title: Never-Ending Behavior-Cloning Agent for Robotic Manipulation
• Authors: Wenqi Liang, Gan Sun, Qian He, Yu Ren, Jiahua Dong, Yang Cong,
• Abstract summary: NBAgent is a language-conditioned Never-ending Behavior-cloning Agent. It learns observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes.
• Score: 38.756955029068294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Relying on multi-modal observations, embodied robots could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents still face existing long-standing challenges, i.e., 3D scene representation and human-level task learning, when adapting into new sequential tasks in practical scenarios. We here investigate these above challenges with NBAgent in embodied robots, a pioneering language-conditioned Never-ending Behavior-cloning Agent. It can continually learn observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-sharedsemantic rendering module and a skill-shared representation distillation module to effectively learn 3D scene semantics from skill-shared attribute, further tackling 3D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like human from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method. Visual results, code, and dataset are provided at: https://neragent.github.io.

Related papers

• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans [58.27029676638521]
  We show how passive human videos can serve as a rich source of data for learning such generalist robots. We learn a human plan predictor that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects.
  arXiv  Detail & Related papers  (2023-12-01T18:54:12Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• XSkill: Cross Embodiment Skill Discovery [41.624343257852146]
  XSkill is an imitation learning framework that discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate skill transfer and composition for unseen tasks.
  arXiv  Detail & Related papers  (2023-07-19T12:51:28Z)
• RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation [33.10577695383743]
  We propose a multi-embodiment, multi-task generalist agent for robotic manipulation called RoboCat. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples.
  arXiv  Detail & Related papers  (2023-06-20T17:35:20Z)
• Surfer: Progressive Reasoning with World Models for Robotic Manipulation [51.26109827779267]
  We introduce a novel and simple robot manipulation framework, called Surfer. Surfer treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene. It is based on the world model, treats robot manipulation as a state transfer of the visual scene, and decouples it into two parts: action and scene.
  arXiv  Detail & Related papers  (2023-06-20T07:06:04Z)
• DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects [8.195608430584073]
  We propose a new benchmark called DexArt, which involves Dexterous manipulation with Articulated objects in a physical simulator. Our main focus is to evaluate the generalizability of the learned policy on unseen articulated objects. We use Reinforcement Learning with 3D representation learning to achieve generalization.
  arXiv  Detail & Related papers  (2023-05-09T18:30:58Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation [55.31301153979621]
  We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2021-09-28T16:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.