Semi-Weakly Supervised Object Kinematic Motion Prediction

• URL: http://arxiv.org/abs/2303.17774v2
• Date: Mon, 3 Apr 2023 02:36:17 GMT
• Title: Semi-Weakly Supervised Object Kinematic Motion Prediction
• Authors: Gengxin Liu, Qian Sun, Haibin Huang, Chongyang Ma, Yulan Guo, Li Yi, Hui Huang, Ruizhen Hu
• Abstract summary: Given a 3D object, kinematic motion prediction aims to identify the mobile parts as well as the corresponding motion parameters. We propose a graph neural network to learn the map between hierarchical part-level segmentation and mobile parts parameters. The network predictions yield a large scale of 3D objects with pseudo labeled mobility information.
• Score: 56.282759127180306
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given a 3D object, kinematic motion prediction aims to identify the mobile parts as well as the corresponding motion parameters. Due to the large variations in both topological structure and geometric details of 3D objects, this remains a challenging task and the lack of large scale labeled data also constrain the performance of deep learning based approaches. In this paper, we tackle the task of object kinematic motion prediction problem in a semi-weakly supervised manner. Our key observations are two-fold. First, although 3D dataset with fully annotated motion labels is limited, there are existing datasets and methods for object part semantic segmentation at large scale. Second, semantic part segmentation and mobile part segmentation is not always consistent but it is possible to detect the mobile parts from the underlying 3D structure. Towards this end, we propose a graph neural network to learn the map between hierarchical part-level segmentation and mobile parts parameters, which are further refined based on geometric alignment. This network can be first trained on PartNet-Mobility dataset with fully labeled mobility information and then applied on PartNet dataset with fine-grained and hierarchical part-level segmentation. The network predictions yield a large scale of 3D objects with pseudo labeled mobility information and can further be used for weakly-supervised learning with pre-existing segmentation. Our experiments show there are significant performance boosts with the augmented data for previous method designed for kinematic motion prediction on 3D partial scans.

Related papers

• SeMoLi: What Moves Together Belongs Together [51.72754014130369]
  We tackle semi-supervised object detection based on motion cues. Recent results suggest that motion-based clustering methods can be used to pseudo-label instances of moving objects. We re-think this approach and suggest that both, object detection, as well as motion-inspired pseudo-labeling, can be tackled in a data-driven manner.
  arXiv  Detail & Related papers  (2024-02-29T18:54:53Z)
• Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation [49.56131393810713]
  We present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs.
  arXiv  Detail & Related papers  (2023-06-08T22:55:32Z)
• Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion [110.84357383258818]
  We propose a novel approach to lift 2D segments to 3D and fuse them by means of a neural field representation. The core of our approach is a slow-fast clustering objective function, which is scalable and well-suited for scenes with a large number of objects. Our approach outperforms the state-of-the-art on challenging scenes from the ScanNet, Hypersim, and Replica datasets.
  arXiv  Detail & Related papers  (2023-06-07T17:57:45Z)
• Unsupervised Kinematic Motion Detection for Part-segmented 3D Shape Collections [14.899075941080541]
  We present an unsupervised approach for discovering articulated motions in a part-segmented 3D shape collection. Our approach is based on a concept we call category closure: any valid articulation of an object's parts should keep the object in the same semantic category. We evaluate our approach by using it to re-discover part motions from the PartNet-Mobility dataset.
  arXiv  Detail & Related papers  (2022-06-17T00:50:36Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)
• Towards Part-Based Understanding of RGB-D Scans [43.4094489272776]
  We propose the task of part-based scene understanding of real-world 3D environments. From an RGB-D scan of a scene, we detect objects, and for each object predict its decomposition into geometric part masks. We leverage an intermediary part graph representation to enable robust completion as well as building of part priors.
  arXiv  Detail & Related papers  (2020-12-03T17:30:02Z)
• Self-Supervised Learning of Part Mobility from Point Cloud Sequence [9.495859862104515]
  We introduce a self-supervised method for segmenting parts and predicting their motion attributes from a point sequence representing a dynamic object. We generate trajectories by using correlations among successive frames of the sequence. We evaluate our method on various tasks including motion part segmentation, motion axis prediction and motion range estimation.
  arXiv  Detail & Related papers  (2020-10-20T11:29:46Z)
• Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection [102.62963605429508]
  Point cloud semantic segmentation plays an essential role in autonomous driving. Current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes. We propose a novel Aware 3D Semantic Detection (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task.
  arXiv  Detail & Related papers  (2020-09-22T14:17:40Z)
• Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-task Baseline [5.000331633798637]
  Moving object segmentation is a crucial task for autonomous vehicles as it can be used to segment objects in a class agnostic manner. Although pixel-wise motion segmentation has been studied in autonomous driving literature, it has been rarely addressed at the instance level. We create a new InstanceMotSeg dataset comprising of 12.9K samples improving upon our KITTIMoSeg dataset.
  arXiv  Detail & Related papers  (2020-08-16T21:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.