NOVUM: Neural Object Volumes for Robust Object Classification

• URL: http://arxiv.org/abs/2305.14668v4
• Date: Wed, 28 Aug 2024 07:28:15 GMT
• Title: NOVUM: Neural Object Volumes for Robust Object Classification
• Authors: Artur Jesslen, Guofeng Zhang, Angtian Wang, Wufei Ma, Alan Yuille, Adam Kortylewski,
• Abstract summary: We show that explicitly integrating 3D compositional object representations into deep networks for image classification leads to a largely enhanced generalization in out-of-distribution scenarios. In particular, we introduce a novel architecture, referred to as NOVUM, that consists of a feature extractor and a neural object volume for every target object class. Our experiments show that NOVUM offers intriguing advantages over standard architectures due to the 3D compositional structure of the object representation.
• Score: 22.411611823528272
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Discriminative models for object classification typically learn image-based representations that do not capture the compositional and 3D nature of objects. In this work, we show that explicitly integrating 3D compositional object representations into deep networks for image classification leads to a largely enhanced generalization in out-of-distribution scenarios. In particular, we introduce a novel architecture, referred to as NOVUM, that consists of a feature extractor and a neural object volume for every target object class. Each neural object volume is a composition of 3D Gaussians that emit feature vectors. This compositional object representation allows for a highly robust and fast estimation of the object class by independently matching the features of the 3D Gaussians of each category to features extracted from an input image. Additionally, the object pose can be estimated via inverse rendering of the corresponding neural object volume. To enable the classification of objects, the neural features at each 3D Gaussian are trained discriminatively to be distinct from (i) the features of 3D Gaussians in other categories, (ii) features of other 3D Gaussians of the same object, and (iii) the background features. Our experiments show that NOVUM offers intriguing advantages over standard architectures due to the 3D compositional structure of the object representation, namely: (1) An exceptional robustness across a spectrum of real-world and synthetic out-of-distribution shifts and (2) an enhanced human interpretability compared to standard models, all while maintaining real-time inference and a competitive accuracy on in-distribution data.

Related papers

• Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space [58.623106094568776]
  3D morphable models (3DMMs) are a powerful tool to represent the possible shapes and appearances of an object category. We introduce a new method, Common3D, that learns 3DMMs of common objects in a fully self-supervised manner from a collection of object-centric videos. Common3D is the first completely self-supervised method that can solve various vision tasks in a zero-shot manner.
  arXiv  Detail & Related papers  (2025-04-30T15:42:23Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction. We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images. We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Escaping Plato's Cave: Robust Conceptual Reasoning through Interpretable 3D Neural Object Volumes [65.63534641857476]
  We introduce CAVE - Concept Aware Volumes for Explanations - a new direction that unifies interpretability and robustness in image classification. We design an inherently-interpretable and robust classifier by extending existing 3D-aware classifiers with concepts extracted from their volumetric representations for classification. In an array of quantitative metrics for interpretability, we compare against different concept-based approaches across the explainable AI literature and show that CAVE discovers well-grounded concepts that are used consistently across images, while achieving superior robustness.
  arXiv  Detail & Related papers  (2025-03-17T17:55:15Z)
• Chirpy3D: Creative Fine-grained 3D Object Fabrication via Part Sampling [128.23917788822948]
  Chirpy3D is a novel approach for fine-grained 3D object generation in a zero-shot setting. The model must infer plausible 3D structures, capture fine-grained details, and generalize to novel objects. Our experiments demonstrate that Chirpy3D surpasses existing methods in generating creative 3D objects with higher quality and fine-grained details.
  arXiv  Detail & Related papers  (2025-01-07T21:14:11Z)
• GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction [70.65250036489128]
  3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene. We propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians. GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption.
  arXiv  Detail & Related papers  (2024-05-27T17:59:51Z)
• OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation [56.028185293563325]
  This paper studies a new open-set problem, the open-vocabulary category-level object pose and size estimation. We first introduce OO3D-9D, a large-scale photorealistic dataset for this task. We then propose a framework built on pre-trained DinoV2 and text-to-image stable diffusion models.
  arXiv  Detail & Related papers  (2024-03-19T03:09:24Z)
• NAP: Neural 3D Articulation Prior [31.875925637190328]
  We propose Neural 3D Articulation Prior (NAP), the first 3D deep generative model to synthesize 3D articulated object models. To generate articulated objects, we first design a novel articulation tree/graph parameterization and then apply a diffusion-denoising probabilistic model over this representation. In order to capture both the geometry and the motion structure whose distribution will affect each other, we design a graph-attention denoising network for learning the reverse diffusion process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:35Z)
• Occupancy Planes for Single-view RGB-D Human Reconstruction [120.5818162569105]
  Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. We propose the occupancy planes (OPlanes) representation, which enables to formulate single-view RGB-D human reconstruction as occupancy prediction on planes which slice through the camera's view frustum.
  arXiv  Detail & Related papers  (2022-08-04T17:59:56Z)
• Category-Agnostic 6D Pose Estimation with Conditional Neural Processes [19.387280883044482]
  We present a novel meta-learning approach for 6D pose estimation on unknown objects. Our algorithm learns object representation in a category-agnostic way, which endows it with strong generalization capabilities across object categories.
  arXiv  Detail & Related papers  (2022-06-14T20:46:09Z)
• Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose [10.028521796737314]
  We study the problem of learning to estimate the 3D object pose from a few labelled examples and a collection of unlabelled data. Our main contribution is a learning framework, neural view synthesis and matching, that can transfer the 3D pose annotation from the labelled to unlabelled images reliably.
  arXiv  Detail & Related papers  (2021-10-27T06:53:53Z)
• DensePose 3D: Lifting Canonical Surface Maps of Articulated Objects to the Third Dimension [71.71234436165255]
  We contribute DensePose 3D, a method that can learn such reconstructions in a weakly supervised fashion from 2D image annotations only. Because it does not require 3D scans, DensePose 3D can be used for learning a wide range of articulated categories such as different animal species. We show significant improvements compared to state-of-the-art non-rigid structure-from-motion baselines on both synthetic and real data on categories of humans and animals.
  arXiv  Detail & Related papers  (2021-08-31T18:33:55Z)
• HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection [39.64891219500416]
  3D object detection methods exploit either voxel-based or point-based features to represent 3D objects in a scene. We introduce in this paper a novel single-stage 3D detection method having the merit of both voxel-based and point-based features.
  arXiv  Detail & Related papers  (2021-04-02T06:34:49Z)
• Unsupervised Learning of 3D Object Categories from Videos in the Wild [75.09720013151247]
  We focus on learning a model from multiple views of a large collection of object instances. We propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:57:01Z)
• Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-supervised category reconstruction [79.98689027127855]
  We propose a new representation of the 3D shape of common object categories that can be learned from a collection of 2D images of independent objects. Our method builds in a novel way on concepts from parametric deformation models, non-parametric 3D reconstruction, and canonical embeddings. It achieves state-of-the-art results in dense 3D reconstruction on public in-the-wild datasets of faces, cars, and birds.
  arXiv  Detail & Related papers  (2020-08-28T15:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.