FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects

• URL: http://arxiv.org/abs/2205.04382v6
• Date: Thu, 2 May 2024 16:17:25 GMT
• Title: FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects
• Authors: Ben Eisner, Harry Zhang, David Held,
• Abstract summary: We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects. We deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.
• Score: 14.034256001448574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore a novel method to perceive and manipulate 3D articulated objects that generalizes to enable a robot to articulate unseen classes of objects. We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects to guide downstream motion planning of the system to articulate the objects. To predict the object motions, we train a neural network to output a dense vector field representing the point-wise motion direction of the points in the point cloud under articulation. We then deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. We train the vision system entirely in simulation, and we demonstrate the capability of our system to generalize to unseen object instances and novel categories in both simulation and the real world, deploying our policy on a Sawyer robot with no finetuning. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.

Related papers

• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• Modular Neural Network Policies for Learning In-Flight Object Catching with a Robot Hand-Arm System [55.94648383147838]
  We present a modular framework designed to enable a robot hand-arm system to learn how to catch flying objects. Our framework consists of five core modules: (i) an object state estimator that learns object trajectory prediction, (ii) a catching pose quality network that learns to score and rank object poses for catching, (iii) a reaching control policy trained to move the robot hand to pre-catch poses, and (iv) a grasping control policy trained to perform soft catching motions. We conduct extensive evaluations of our framework in simulation for each module and the integrated system, to demonstrate high success rates of in-flight
  arXiv  Detail & Related papers  (2023-12-21T16:20:12Z)
• Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs [53.66070434419739]
  Generalizable articulated object manipulation is essential for home-assistant robots. We propose a kinematic-aware prompting framework that prompts Large Language Models with kinematic knowledge of objects to generate low-level motion waypoints. Our framework outperforms traditional methods on 8 categories seen and shows a powerful zero-shot capability for 8 unseen articulated object categories.
  arXiv  Detail & Related papers  (2023-11-06T03:26:41Z)
• DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations [51.87067543670535]
  We propose a robot-learning system that can take a small number of human demonstrations and learn to grasp unseen object poses. We train a dexterous grasping policy that takes the point clouds of the object as input and predicts continuous actions to grasp objects from different initial robot states. The policy learned from our dataset can generalize well on unseen object poses in both simulation and the real world.
  arXiv  Detail & Related papers  (2022-09-28T17:51:49Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Learning to Infer Kinematic Hierarchies for Novel Object Instances [12.50766181856788]
  Our system infers the moving parts of an object and the kinematic couplings that relate them. We evaluate our system on simulated scans of 3D objects, andwe demonstrate a proof-of-concept use of our system to drivereal-world robotic manipulation.
  arXiv  Detail & Related papers  (2021-10-15T07:50:48Z)
• 3D Neural Scene Representations for Visuomotor Control [78.79583457239836]
  We learn models for dynamic 3D scenes purely from 2D visual observations. A dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks.
  arXiv  Detail & Related papers  (2021-07-08T17:49:37Z)
• VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects [19.296344218177534]
  The space of 3D articulated objects is exceptionally rich in their myriad semantic categories, diverse shape geometry, and complicated part functionality. Previous works mostly abstract kinematic structure with estimated joint parameters and part poses as the visual representations for manipulating 3D articulated objects. We propose object-centric actionable visual priors as a novel perception-interaction handshaking point that the perception system outputs more actionable guidance than kinematic structure estimation.
  arXiv  Detail & Related papers  (2021-06-28T07:47:31Z)
• A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects [25.428781562909606]
  We present a framework for solving long-horizon planning problems involving manipulation of rigid objects. Our method plans in the space of object subgoals and frees the planner from reasoning about robot-object interaction dynamics.
  arXiv  Detail & Related papers  (2020-11-16T18:59:33Z)
• Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction [24.72947291987545]
  Key challenge for an agent learning to interact with the world is to reason about physical properties of objects. We propose a novel approach for modeling the dynamics of a robot's interactions directly from unlabeled 3D point clouds and images.
  arXiv  Detail & Related papers  (2020-08-02T11:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.