Surface-Aware Distilled 3D Semantic Features

• URL: http://arxiv.org/abs/2503.18254v1
• Date: Mon, 24 Mar 2025 00:36:16 GMT
• Title: Surface-Aware Distilled 3D Semantic Features
• Authors: Lukas Uzolas, Elmar Eisemann, Petr Kellnhofer,
• Abstract summary: Many 3D tasks such as pose alignment, animation, motion transfer, and 3D reconstruction rely on establishing correspondences between 3D shapes.<n>This challenge has recently been approached by matching of semantic features from pre-trained vision models.<n>We learn a surface-aware embedding space that is robust to these ambiguities.
• Score: 10.263762787854862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many 3D tasks such as pose alignment, animation, motion transfer, and 3D reconstruction rely on establishing correspondences between 3D shapes. This challenge has recently been approached by matching of semantic features from pre-trained vision models. However, despite their power, these features struggle to differentiate instances of the same semantic class such as "left hand" versus "right hand" which leads to substantial mapping errors. To solve this, we learn a surface-aware embedding space that is robust to these ambiguities. Importantly, our approach is self-supervised and requires only a small number of unpaired training meshes to infer features for new 3D shapes at test time. We achieve this by introducing a contrastive loss that preserves the semantic content of the features distilled from foundational models while disambiguating features located far apart on the shape's surface. We observe superior performance in correspondence matching benchmarks and enable downstream applications including in-part segmentation, pose alignment, and motion transfer. The project site is available at https://lukas.uzolas.com/SurfaceAware3DFeaturesSite.

Related papers

• Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning [10.51523800391153]
  We evaluate and enhance the 3D awareness of ViT-based models. We propose a simple yet effective finetuning strategy based on 3D correspondences.
  arXiv  Detail & Related papers  (2024-11-29T04:02:11Z)
• Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping [34.98831146003579]
  One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. We propose the textitneural attention field for representing semantic-aware dense feature fields in the 3D space.
  arXiv  Detail & Related papers  (2024-10-30T14:06:51Z)
• MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors [11.118490283303407]
  We propose a neural field semantic reconstruction approach to lift inferred image-level noisy priors to 3D. Our method produces accurate semantics and geometry in both 3D and 2D space.
  arXiv  Detail & Related papers  (2024-09-21T05:12:13Z)
• PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models [51.24979014650188]
  We present PointSeg, a training-free paradigm that leverages off-the-shelf vision foundation models to address 3D scene perception tasks. PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to align their corresponding pixels across frames. Our approach significantly surpasses the state-of-the-art specialist training-free model by 14.1$%$, 12.3$%$, and 12.6$%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets.
  arXiv  Detail & Related papers  (2024-03-11T03:28:20Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [114.47216525866435]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• Neural Semantic Surface Maps [52.61017226479506]
  We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement.
  arXiv  Detail & Related papers  (2023-09-09T16:21:56Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose [10.028521796737314]
  We study the problem of learning to estimate the 3D object pose from a few labelled examples and a collection of unlabelled data. Our main contribution is a learning framework, neural view synthesis and matching, that can transfer the 3D pose annotation from the labelled to unlabelled images reliably.
  arXiv  Detail & Related papers  (2021-10-27T06:53:53Z)
• DensePose 3D: Lifting Canonical Surface Maps of Articulated Objects to the Third Dimension [71.71234436165255]
  We contribute DensePose 3D, a method that can learn such reconstructions in a weakly supervised fashion from 2D image annotations only. Because it does not require 3D scans, DensePose 3D can be used for learning a wide range of articulated categories such as different animal species. We show significant improvements compared to state-of-the-art non-rigid structure-from-motion baselines on both synthetic and real data on categories of humans and animals.
  arXiv  Detail & Related papers  (2021-08-31T18:33:55Z)
• Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes [86.2129580231191]
  Adjoint Rigid Transform (ART) Network is a neural module which can be integrated with a variety of 3D networks. ART learns to rotate input shapes to a learned canonical orientation, which is crucial for a lot of tasks. We will release our code and pre-trained models for further research.
  arXiv  Detail & Related papers  (2021-02-01T20:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.