Learning Dense Feature Matching via Lifting Single 2D Image to 3D Space

• URL: http://arxiv.org/abs/2507.00392v2
• Date: Sat, 05 Jul 2025 23:13:08 GMT
• Title: Learning Dense Feature Matching via Lifting Single 2D Image to 3D Space
• Authors: Yingping Liang, Yutao Hu, Wenqi Shao, Ying Fu,
• Abstract summary: We propose a novel two-stage framework that lifts 2D images to 3D space, taking full advantage of large-scale and diverse single-view images.<n>In the first stage, we learn a 3D-aware feature encoder using a combination of multi-view image synthesis and 3D feature Gaussian representation.<n>In the second stage, a novel-view rendering strategy, combined with large-scale synthetic data generation from single-view images, is employed to learn a feature decoder.
• Score: 17.603217168518356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Feature matching plays a fundamental role in many computer vision tasks, yet existing methods heavily rely on scarce and clean multi-view image collections, which constrains their generalization to diverse and challenging scenarios. Moreover, conventional feature encoders are typically trained on single-view 2D images, limiting their capacity to capture 3D-aware correspondences. In this paper, we propose a novel two-stage framework that lifts 2D images to 3D space, named as \textbf{Lift to Match (L2M)}, taking full advantage of large-scale and diverse single-view images. To be specific, in the first stage, we learn a 3D-aware feature encoder using a combination of multi-view image synthesis and 3D feature Gaussian representation, which injects 3D geometry knowledge into the encoder. In the second stage, a novel-view rendering strategy, combined with large-scale synthetic data generation from single-view images, is employed to learn a feature decoder for robust feature matching, thus achieving generalization across diverse domains. Extensive experiments demonstrate that our method achieves superior generalization across zero-shot evaluation benchmarks, highlighting the effectiveness of the proposed framework for robust feature matching.

Related papers

• CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction [25.468907201804093]
  Large Reconstruction Models (LRMs) have shown great promise in leveraging multi-view images generated by 2D diffusion models to extract 3D content.<n>However, 2D diffusion models often struggle to produce dense images with strong multi-view consistency.<n>We present CDI3D, a feed-forward framework designed for efficient, high-quality image-to-3D generation with view.
  arXiv  Detail & Related papers  (2025-03-11T03:08:43Z)
• GSemSplat: Generalizable Semantic 3D Gaussian Splatting from Uncalibrated Image Pairs [33.74118487769923]
  We introduce GSemSplat, a framework that learns semantic representations linked to 3D Gaussians without per-scene optimization, dense image collections or calibration.<n>We employ a dual-feature approach that leverages both region-specific and context-aware semantic features as supervision in the 2D space.
  arXiv  Detail & Related papers  (2024-12-22T09:06:58Z)
• Refine3DNet: Scaling Precision in 3D Object Reconstruction from Multi-View RGB Images using Attention [2.037112541541094]
  We introduce a hybrid strategy featuring a visual auto-encoder with self-attention mechanisms and a 3D refiner network.<n>Our approach, combined with JTSO, outperforms state-of-the-art techniques in single and multi-view 3D reconstruction.
  arXiv  Detail & Related papers  (2024-12-01T08:53:39Z)
• 3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance [61.06034736050515]
  We introduce a method capable of generating camera-controlled viewpoints from a single input image.<n>Our method excels in handling complex and diverse scenes without extensive training or additional 3D and multiview data.
  arXiv  Detail & Related papers  (2024-08-12T13:53:40Z)
• DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields [68.94868475824575]
  This paper introduces a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations. We leverage the strong semantic prior within a 3D generative model to train a semantic decoder. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data.
  arXiv  Detail & Related papers  (2023-11-18T21:58:28Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [111.16358607889609]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation.<n>For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• Multiview Compressive Coding for 3D Reconstruction [77.95706553743626]
  We introduce a simple framework that operates on 3D points of single objects or whole scenes. Our model, Multiview Compressive Coding, learns to compress the input appearance and geometry to predict the 3D structure.
  arXiv  Detail & Related papers  (2023-01-19T18:59:52Z)
• Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis [48.33860286920389]
  3D-aware image synthesis aims to generate images of objects from multiple views by learning a 3D representation. Existing approaches lack geometry constraints, hence usually fail to generate multi-view consistent images. We propose Multi-View Consistent Generative Adrial Networks (MVCGAN) for high-quality 3D-aware image synthesis with geometry constraints.
  arXiv  Detail & Related papers  (2022-04-13T11:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.