Dense Semantic Matching with VGGT Prior

• URL: http://arxiv.org/abs/2509.21263v1
• Date: Thu, 25 Sep 2025 14:56:11 GMT
• Title: Dense Semantic Matching with VGGT Prior
• Authors: Songlin Yang, Tianyi Wei, Yushi Lan, Zeqi Xiao, Anyi Rao, Xingang Pan,
• Abstract summary: We propose an approach that retains VGGT's intrinsic strengths by reusing early feature stages, fine-tuning later ones, and adding a semantic head for bidirectional correspondences.<n>Our approach achieves superior geometry awareness, matching reliability, and manifold preservation, outperforming previous baselines.
• Score: 49.42199006453071
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Semantic matching aims to establish pixel-level correspondences between instances of the same category and represents a fundamental task in computer vision. Existing approaches suffer from two limitations: (i) Geometric Ambiguity: Their reliance on 2D foundation model features (e.g., Stable Diffusion, DINO) often fails to disambiguate symmetric structures, requiring extra fine-tuning yet lacking generalization; (ii) Nearest-Neighbor Rule: Their pixel-wise matching ignores cross-image invisibility and neglects manifold preservation. These challenges call for geometry-aware pixel descriptors and holistic dense correspondence mechanisms. Inspired by recent advances in 3D geometric foundation models, we turn to VGGT, which provides geometry-grounded features and holistic dense matching capabilities well aligned with these needs. However, directly transferring VGGT is challenging, as it was originally designed for geometry matching within cross views of a single instance, misaligned with cross-instance semantic matching, and further hindered by the scarcity of dense semantic annotations. To address this, we propose an approach that (i) retains VGGT's intrinsic strengths by reusing early feature stages, fine-tuning later ones, and adding a semantic head for bidirectional correspondences; and (ii) adapts VGGT to the semantic matching scenario under data scarcity through cycle-consistent training strategy, synthetic data augmentation, and progressive training recipe with aliasing artifact mitigation. Extensive experiments demonstrate that our approach achieves superior geometry awareness, matching reliability, and manifold preservation, outperforming previous baselines.

Related papers

• Inverse Rendering for High-Genus 3D Surface Meshes from Multi-view Images with Persistent Homology Priors [11.227213428407673]
  Reconstructing 3D objects from images is inherently an ill-posed problem due to ambiguities in geometry, appearance, and topology.<n>This paper introduces collaborative rendering with persistent homology priors, a novel strategy that leverages topological constraints to resolve these ambiguities.
  arXiv  Detail & Related papers  (2026-01-17T20:06:19Z)
• IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction [82.53307702809606]
  Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions.<n>We propose InstanceGrounded Geometry Transformer (IGGT) to unify the knowledge for both spatial reconstruction and instance-level contextual understanding.
  arXiv  Detail & Related papers  (2025-10-26T14:57:44Z)
• AlignGS: Aligning Geometry and Semantics for Robust Indoor Reconstruction from Sparse Views [18.361136390711415]
  The demand for semantically rich 3D models of indoor scenes is rapidly growing, driven by applications in augmented reality, virtual reality, and robotics.<n>Existing methods often treat semantics as a passive feature painted on an already-formed, and potentially flawed, geometry.<n>This paper introduces AlignGS, a novel framework that actualizes this vision by pioneering a synergistic, end-to-end optimization of geometry and semantics.
  arXiv  Detail & Related papers  (2025-10-09T06:30:20Z)
• H3R: Hybrid Multi-view Correspondence for Generalizable 3D Reconstruction [39.22287224290769]
  H3R is a hybrid framework that integrates latent fusion with attention-based feature aggregation.<n>By integrating both paradigms, our approach enhances generalization while converging 2$times$ faster than existing methods.<n>Our method supports variable-number and high-resolution input views while demonstrating robust cross-dataset generalization.
  arXiv  Detail & Related papers  (2025-08-05T05:56:30Z)
• Geometry-Editable and Appearance-Preserving Object Compositon [67.98806888489385]
  General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties.<n>Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation.<n>We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion model that first leverages semantic embeddings to implicitly capture desired geometric transformations.
  arXiv  Detail & Related papers  (2025-05-27T09:05:28Z)
• Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction [61.484280369655536]
  Camera-based 3D Semantic Occupancy Prediction (SOP) is crucial for understanding complex 3D scenes from limited 2D image observations.<n>Existing SOP methods typically aggregate contextual features to assist the occupancy representation learning.<n>We introduce a new Hierarchical context alignment paradigm for a more accurate SOP (Hi-SOP)
  arXiv  Detail & Related papers  (2024-12-11T09:53:10Z)
• SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generation [122.47961178994456]
  SeMv-3D is a novel framework that jointly enhances semantic alignment and multi-view consistency in GT23D generation.<n>At its core, we introduce Triplane Prior Learning (TPL), which effectively learns triplane priors.<n>We also present Prior-based Semantic Aligning in Triplanes (SAT), which enables consistent any-view synthesis.
  arXiv  Detail & Related papers  (2024-10-10T07:02:06Z)
• Parallel Vertex Diffusion for Unified Visual Grounding [38.94276071029081]
  Unified visual grounding pursues a simple and generic technical route to leverage multi-task data with less task-specific design. Most advanced methods typically present boxes and masks as a sequence to model referring detection and segmentation.
  arXiv  Detail & Related papers  (2023-03-13T15:51:38Z)
• Self-supervised Geometric Perception [96.89966337518854]
  Self-supervised geometric perception is a framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels. We show that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.
  arXiv  Detail & Related papers  (2021-03-04T15:34:43Z)
• Deep Semantic Matching with Foreground Detection and Cycle-Consistency [103.22976097225457]
  We address weakly supervised semantic matching based on a deep network. We explicitly estimate the foreground regions to suppress the effect of background clutter. We develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent.
  arXiv  Detail & Related papers  (2020-03-31T22:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.