Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2411.03555v1
• Date: Tue, 05 Nov 2024 23:28:57 GMT
• Title: Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting
• Authors: Michael Büttner, Jonathan Francis, Helge Rhodin, Andrew Melnik,
• Abstract summary: This paper introduces a method to enhance Interactive Imitation Learning (IIL) by extracting touch interaction points and tracking object movement from video demonstrations. The approach extends current IIL systems by providing robots with detailed knowledge of both where and how to interact with objects, particularly complex articulated ones like doors and drawers.
• Score: 17.03927416536173
• License:
• Abstract: This paper introduces a method to enhance Interactive Imitation Learning (IIL) by extracting touch interaction points and tracking object movement from video demonstrations. The approach extends current IIL systems by providing robots with detailed knowledge of both where and how to interact with objects, particularly complex articulated ones like doors and drawers. By leveraging cutting-edge techniques such as 3D Gaussian Splatting and FoundationPose for tracking, this method allows robots to better understand and manipulate objects in dynamic environments. The research lays the foundation for more effective task learning and execution in autonomous robotic systems.

Related papers

• Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking [59.87033229815062]
  Articulated object manipulation requires precise object interaction, where the object's axis must be carefully considered. Previous research employed interactive perception for manipulating articulated objects, but typically, open-loop approaches often suffer from overlooking the interaction dynamics. We present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
  arXiv  Detail & Related papers  (2024-09-24T17:59:56Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• Cognitive Planning for Object Goal Navigation using Generative AI Models [0.979851640406258]
  We present a novel framework for solving the object goal navigation problem that generates efficient exploration strategies. Our approach enables a robot to navigate unfamiliar environments by leveraging Large Language Models (LLMs) and Large Vision-Language Models (LVLMs)
  arXiv  Detail & Related papers  (2024-03-30T10:54:59Z)
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models [38.503337052122234]
  Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation. We aim to synthesize robot trajectories for a variety of manipulation tasks given an open-set of instructions and an open-set of objects. We demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions.
  arXiv  Detail & Related papers  (2023-07-12T07:40:48Z)
• TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM [0.0]
  3D object tracking is a critical task in autonomous driving systems. In this paper we investigate the use of triplet embeddings in combination with motion representations for 3D object tracking.
  arXiv  Detail & Related papers  (2022-10-28T15:23:50Z)
• Silver-Bullet-3D at ManiSkill 2021: Learning-from-Demonstrations and Heuristic Rule-based Methods for Object Manipulation [118.27432851053335]
  This paper presents an overview and comparative analysis of our systems designed for the following two tracks in SAPIEN ManiSkill Challenge 2021: No Interaction Track. The No Interaction track targets for learning policies from pre-collected demonstration trajectories. In this track, we design a Heuristic Rule-based Method (HRM) to trigger high-quality object manipulation by decomposing the task into a series of sub-tasks. For each sub-task, the simple rule-based controlling strategies are adopted to predict actions that can be applied to robotic arms.
  arXiv  Detail & Related papers  (2022-06-13T16:20:42Z)
• FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects [14.034256001448574]
  We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects. We deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.
  arXiv  Detail & Related papers  (2022-05-09T15:35:33Z)
• Learnable Online Graph Representations for 3D Multi-Object Tracking [156.58876381318402]
  We propose a unified and learning based approach to the 3D MOT problem. We employ a Neural Message Passing network for data association that is fully trainable. We show the merit of the proposed approach on the publicly available nuScenes dataset by achieving state-of-the-art performance of 65.6% AMOTA and 58% fewer ID-switches.
  arXiv  Detail & Related papers  (2021-04-23T17:59:28Z)
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators [4.317864702902075]
  We present the first RL-based system for a mobile manipulator that can (a) achieve targeted grasping generalizing to unseen target objects, (b) learn complex grasping strategies for cluttered scenes with occluded objects, and (c) perform active vision through its movable wrist camera to better locate objects. We train and evaluate our system in a simulated environment, identify key components for improving performance, analyze its behaviors, and transfer to a real-world setup.
  arXiv  Detail & Related papers  (2020-07-16T02:47:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.