Real-time Detection of 2D Tool Landmarks with Synthetic Training Data

• URL: http://arxiv.org/abs/2210.11991v1
• Date: Fri, 21 Oct 2022 14:31:43 GMT
• Title: Real-time Detection of 2D Tool Landmarks with Synthetic Training Data
• Authors: Bram Vanherle, Jeroen Put, Nick Michiels, Frank Van Reeth
• Abstract summary: In this paper a deep learning architecture is presented that can, in real time, detect the 2D locations of certain landmarks of physical tools, such as a hammer or screwdriver. To avoid the labor of manual labeling, the network is trained on synthetically generated data. It is shown that the model presented in this paper, named Intermediate Heatmap Model (IHM), generalizes to real images when trained on synthetic data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper a deep learning architecture is presented that can, in real time, detect the 2D locations of certain landmarks of physical tools, such as a hammer or screwdriver. To avoid the labor of manual labeling, the network is trained on synthetically generated data. Training computer vision models on computer generated images, while still achieving good accuracy on real images, is a challenge due to the difference in domain. The proposed method uses an advanced rendering method in combination with transfer learning and an intermediate supervision architecture to address this problem. It is shown that the model presented in this paper, named Intermediate Heatmap Model (IHM), generalizes to real images when trained on synthetic data. To avoid the need for an exact textured 3D model of the tool in question, it is shown that the model will generalize to an unseen tool when trained on a set of different 3D models of the same type of tool. IHM is compared to two existing approaches to keypoint detection and it is shown that it outperforms those at detecting tool landmarks, trained on synthetic data.

Related papers

• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• Towards Fair and Comprehensive Comparisons for Image-Based 3D Object Detection [73.32210225999056]
  We build a module-designed and formulate unified training standards for 3D object detection. We also design an error diagnosis toolbox to measure the detailed characterization of detection models.
  arXiv  Detail & Related papers  (2023-10-09T06:43:48Z)
• Robust Category-Level 3D Pose Estimation from Synthetic Data [17.247607850702558]
  We introduce SyntheticP3D, a new synthetic dataset for object pose estimation generated from CAD models. We propose a novel approach (CC3D) for training neural mesh models that perform pose estimation via inverse rendering.
  arXiv  Detail & Related papers  (2023-05-25T14:56:03Z)
• RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects [68.85305626324694]
  Ray-marching in Camera Space (RiCS) is a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We show that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects.
  arXiv  Detail & Related papers  (2022-05-14T05:35:35Z)
• Learning Dense Correspondence from Synthetic Environments [27.841736037738286]
  Existing methods map manually labelled human pixels in real 2D images onto the 3D surface, which is prone to human error. We propose to solve the problem of data scarcity by training 2D-3D human mapping algorithms using automatically generated synthetic data.
  arXiv  Detail & Related papers  (2022-03-24T08:13:26Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• Zero in on Shape: A Generic 2D-3D Instance Similarity Metric learned from Synthetic Data [3.71630298053787]
  We present a network architecture which compares RGB images and untextured 3D models by the similarity of the represented shape. Our system is optimised for zero-shot retrieval, meaning it can recognise shapes never shown in training.
  arXiv  Detail & Related papers  (2021-08-09T14:44:08Z)
• Leveraging 2D Data to Learn Textured 3D Mesh Generation [33.32377849866736]
  We present the first generative model of textured 3D meshes. We train our model to explain a distribution of images by modelling each image as a 3D foreground object. It learns to generate meshes that when rendered, produce images similar to those in its training set.
  arXiv  Detail & Related papers  (2020-04-08T18:00:37Z)
• Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data [77.34069717612493]
  We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass.
  arXiv  Detail & Related papers  (2020-03-21T03:51:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.