SEMAGIC: Learning Semantically Consistent Deformable 3D Representations from In-the-Wild Images

• URL: http://arxiv.org/abs/2605.27938v1
• Date: Wed, 27 May 2026 04:18:06 GMT
• Title: SEMAGIC: Learning Semantically Consistent Deformable 3D Representations from In-the-Wild Images
• Abstract summary: Learning deformable 3D object models from single-view in-the-wild images has enabled impressive 3D shape reconstruction without supervision.<n>We introduce a framework for learning semantically consistent deformable 3D representations from single-view in-the-wild images.
• Score: 32.19024127550901
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning deformable 3D object models from single-view in-the-wild images has enabled impressive 3D shape reconstruction without supervision. However, it remains unclear whether these models capture the semantic structure required for downstream tasks. We find that existing deformable reconstruction approaches, despite producing visually plausible geometry, yield unstable correspondences across instances and perform poorly on semantic correspondence benchmarks. We introduce SEMAGIC, a framework for learning semantically consistent deformable 3D representations from single-view in-the-wild images. Rather than treating reconstruction as the end goal, SEMAGIC uses deformable modeling as a mechanism to discover category-level correspondences. Each category is represented by a canonical template mesh and a learned deformation field, functioning similarly to an autoencoder that reconstructs instance geometry from image features, enabling vertices to maintain consistent semantic meaning across instances. Semantic consistency is enforced during training through (i) a feature-level consistency loss aligning semantic features between canonical and deformed meshes, and (ii) vertex-index-conditioned deformation that preserves semantic correspondence across instances. By explicitly coupling geometric deformation with semantic alignment, SEMAGIC produces representations that maintain stable part correspondences across intra-category variation. Experiments demonstrate that SEMAGIC improves semantic correspondence of deformable models by +14.7 PCK@0.1 on SPair-71k, establishing deformable models as effective semantic 3D representations.

Related papers

• SemDynReg: Semantics-Guided Deformation Regularization for Dynamic 3D Gaussian Splatting [1.1934335703226404]
  We introduce a semantics-guided framework that enforces dynamic regularization at the object level.<n>We achieve higher PSNR and SSIM and lower LPIPS in dynamic 3DGS rendering.
  arXiv  Detail & Related papers  (2026-06-27T00:18:55Z)
• SCAPO: Self-Supervised Category-Level Articulated Pose Estimation from a Single 3D Observation [54.98864436812187]
  We propose a self-supervised framework that estimates canonical geometry, rigid part segmentation, and joint pivots, axes, and articulation states from a single RGB-D observation.<n> Experiments on synthetic and real articulated-object datasets show that our SCAPO recovers consistent part structure and accurate articulation parameters and outperforms all self-supervised baselines.
  arXiv  Detail & Related papers  (2026-06-01T09:04:49Z)
• Matching Shapes Under Different Topologies: A Topology-Adaptive Deformation Guided Approach [6.851721795186258]
  Non-rigid 3D mesh matching is a critical step in computer vision and computer graphics pipelines.<n>We tackle matching meshes that contain topological artefacts which can break the assumption made by current approaches.<n>We are motivated by real-world scenarios such as per-frame multi-view reconstructions, often suffering from topological artefacts.<n>We show that, while not relying on any data-driven prior, our approach applies to highly non-isometric shapes and shapes with topological artefacts, including noisy per-frame multi-view reconstructions, even outperforming methods trained on large datasets in 3D alignment quality.
  arXiv  Detail & Related papers  (2025-09-08T16:29:44Z)
• DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field [29.42222066097076]
  Estimating 6D poses and reconstructing 3D shapes of objects in open-world scenes from RGB-depth image pairs is challenging. We propose the DTF-Net, a novel framework for pose estimation and shape reconstruction based on implicit neural fields of object categories.
  arXiv  Detail & Related papers  (2023-08-04T10:35:40Z)
• Semantic Validation in Structure from Motion [0.0]
  Structure from Motion (SfM) is the process of recovering the 3D structure of a scene from a series of projective measurements. SfM consists of three main steps; feature detection and matching, camera motion estimation, and recovery of 3D structure. This project offers a novel method for improved validation of 3D SfM models.
  arXiv  Detail & Related papers  (2023-04-05T12:58:59Z)
• Self-Supervised Image Representation Learning with Geometric Set Consistency [50.12720780102395]
  We propose a method for self-supervised image representation learning under the guidance of 3D geometric consistency. Specifically, we introduce 3D geometric consistency into a contrastive learning framework to enforce the feature consistency within image views.
  arXiv  Detail & Related papers  (2022-03-29T08:57:33Z)
• 3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow [61.62796058294777]
  Reconstructing 3D shape from a single 2D image is a challenging task. Most of the previous methods still struggle to extract semantic attributes for 3D reconstruction task. We propose 3DAttriFlow to disentangle and extract semantic attributes through different semantic levels in the input images.
  arXiv  Detail & Related papers  (2022-03-29T02:03:31Z)
• Learning Canonical 3D Object Representation for Fine-Grained Recognition [77.33501114409036]
  We propose a novel framework for fine-grained object recognition that learns to recover object variation in 3D space from a single image. We represent an object as a composition of 3D shape and its appearance, while eliminating the effect of camera viewpoint. By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object.
  arXiv  Detail & Related papers  (2021-08-10T12:19:34Z)
• Deep Implicit Templates for 3D Shape Representation [70.9789507686618]
  We propose a new 3D shape representation that supports explicit correspondence reasoning in deep implicit representations. Our key idea is to formulate DIFs as conditional deformations of a template implicit function. We show that our method can not only learn a common implicit template for a collection of shapes, but also establish dense correspondences across all the shapes simultaneously without any supervision.
  arXiv  Detail & Related papers  (2020-11-30T06:01:49Z)
• Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-supervised category reconstruction [79.98689027127855]
  We propose a new representation of the 3D shape of common object categories that can be learned from a collection of 2D images of independent objects. Our method builds in a novel way on concepts from parametric deformation models, non-parametric 3D reconstruction, and canonical embeddings. It achieves state-of-the-art results in dense 3D reconstruction on public in-the-wild datasets of faces, cars, and birds.
  arXiv  Detail & Related papers  (2020-08-28T15:44:05Z)
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency [142.71430568330172]
  We learn a self-supervised, single-view 3D reconstruction model that predicts the shape, texture and camera pose of a target object. The proposed method does not necessitate 3D supervision, manually annotated keypoints, multi-view images of an object or a prior 3D template.
  arXiv  Detail & Related papers  (2020-03-13T20:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.