Disentangling 3D Prototypical Networks For Few-Shot Concept Learning

• URL: http://arxiv.org/abs/2011.03367v3
• Date: Tue, 20 Jul 2021 19:07:01 GMT
• Title: Disentangling 3D Prototypical Networks For Few-Shot Concept Learning
• Authors: Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao-Yu Tung, Adam W Harley, Katerina Fragkiadaki
• Abstract summary: We present neural architectures that disentangle RGB-D images into objects' shapes and styles and a map of the background scene. Our networks incorporate architectural biases that reflect the image formation process, 3D geometry of the world scene, and shape-style interplay.
• Score: 29.02523358573336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present neural architectures that disentangle RGB-D images into objects' shapes and styles and a map of the background scene, and explore their applications for few-shot 3D object detection and few-shot concept classification. Our networks incorporate architectural biases that reflect the image formation process, 3D geometry of the world scene, and shape-style interplay. They are trained end-to-end self-supervised by predicting views in static scenes, alongside a small number of 3D object boxes. Objects and scenes are represented in terms of 3D feature grids in the bottleneck of the network. We show that the proposed 3D neural representations are compositional: they can generate novel 3D scene feature maps by mixing object shapes and styles, resizing and adding the resulting object 3D feature maps over background scene feature maps. We show that classifiers for object categories, color, materials, and spatial relationships trained over the disentangled 3D feature sub-spaces generalize better with dramatically fewer examples than the current state-of-the-art, and enable a visual question answering system that uses them as its modules to generalize one-shot to novel objects in the scene.

Related papers

• Disentangled 3D Scene Generation with Layout Learning [109.03233745767062]
  We introduce a method to generate 3D scenes that are disentangled into their component objects. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. We show that despite its simplicity, our approach successfully generates 3D scenes into individual objects.
  arXiv  Detail & Related papers  (2024-02-26T18:54:15Z)
• Generating Visual Spatial Description via Holistic 3D Scene Understanding [88.99773815159345]
  Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images. We construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes.
  arXiv  Detail & Related papers  (2023-05-19T15:53:56Z)
• Graphics Capsule: Learning Hierarchical 3D Face Representations from 2D Images [82.5266467869448]
  We propose an Inverse Graphics Capsule Network (IGC-Net) to learn the hierarchical 3D face representations from large-scale unlabeled images. IGC-Net first decomposes the objects into a set of semantic-consistent part-level descriptions and then assembles them into object-level descriptions to build the hierarchy.
  arXiv  Detail & Related papers  (2023-03-20T06:32:55Z)
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories [69.29406107513621]
  Single-view 3D Mesh Reconstruction is a fundamental computer vision task that aims at recovering 3D shapes from single-view RGB images. This paper tackles Single-view 3D Mesh Reconstruction, to study the model generalization on unseen categories. We propose an end-to-end two-stage network, GenMesh, to break the category boundaries in reconstruction.
  arXiv  Detail & Related papers  (2022-08-04T14:13:35Z)
• Neural Groundplans: Persistent Neural Scene Representations from a Single Image [90.04272671464238]
  We present a method to map 2D image observations of a scene to a persistent 3D scene representation. We propose conditional neural groundplans as persistent and memory-efficient scene representations.
  arXiv  Detail & Related papers  (2022-07-22T17:41:24Z)
• 3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations [29.61554189447989]
  We propose a system that learns to detect objects and infer their 3D poses in RGB-D images. Many existing systems can identify objects and infer 3D poses, but they heavily rely on human labels and 3D annotations.
  arXiv  Detail & Related papers  (2020-10-30T13:56:09Z)
• Learning to Reconstruct and Segment 3D Objects [4.709764624933227]
  We aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks. This thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding.
  arXiv  Detail & Related papers  (2020-10-19T15:09:04Z)
• GRF: Learning a General Radiance Field for 3D Representation and Rendering [4.709764624933227]
  We present a simple yet powerful neural network that implicitly represents and renders 3D objects and scenes only from 2D observations. The network models 3D geometries as a general radiance field, which takes a set of 2D images with camera poses and intrinsics as input. Our method can generate high-quality and realistic novel views for novel objects, unseen categories and challenging real-world scenes.
  arXiv  Detail & Related papers  (2020-10-09T14:21:43Z)
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors [69.02332607843569]
  PriSMONet is a novel approach for learning Multi-Object 3D scene decomposition and representations from single images. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.
  arXiv  Detail & Related papers  (2020-10-08T14:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.