Repurposing Geometric Foundation Models for Multi-view Diffusion

• URL: http://arxiv.org/abs/2603.22275v1
• Date: Mon, 23 Mar 2026 17:57:05 GMT
• Title: Repurposing Geometric Foundation Models for Multi-view Diffusion
• Abstract summary: We propose a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion.<n>Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space.
• Score: 54.65956050951508
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent advances in generative latent spaces have driven substantial progress in single-image generation, the optimal latent space for novel view synthesis (NVS) remains largely unexplored. In particular, NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in a view-independent VAE latent space. In this paper, we propose Geometric Latent Diffusion (GLD), a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion. We show that these features not only support high-fidelity RGB reconstruction but also encode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space. Notably, GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite training its diffusion model from scratch without such generative pretraining.

Related papers

• SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness [62.38746300197673]
  We propose a novel framework that internalizes 3D geometric awareness into unified image generation models.<n>Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer.<n>By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts.<n>These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter.
  arXiv  Detail & Related papers  (2026-04-29T06:46:59Z)
• VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward [14.297118875382871]
  VGGRPO is a latent geometry-guided framework for geometry-aware video post-training.<n>It stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space.<n>It improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding.
  arXiv  Detail & Related papers  (2026-03-27T16:57:51Z)
• TIMI: Training-Free Image-to-3D Multi-Instance Generation with Spatial Fidelity [126.89088014048001]
  We propose a Training-free framework for Image-to-3D Multi-Instance generation that achieves high spatial fidelity.<n>Specifically, we first introduce an Instance-aware Separation Guidance (ISG) module, which facilitates instance disentanglement.<n>We then devise a Spatial-adaptive Geometry-stabilized Update (SGU) module that promotes the preservation of the geometric characteristics of instances.
  arXiv  Detail & Related papers  (2026-03-02T02:10:28Z)
• Joint Geometry-Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion [57.09673862519791]
  This paper introduces textbfJGA-LBD, a novel framework that unifies the modeling of geometry and appearance into a joint latent representation.<n> Experiments demonstrate that JGA-LBD outperforms current state-of-the-art approaches in terms of both geometry fidelity and appearance quality.
  arXiv  Detail & Related papers  (2026-01-01T12:48:56Z)
• SVRecon: Sparse Voxel Rasterization for Surface Reconstruction [60.92372415355283]
  We extend the recently proposed sparse voxelization paradigm to the task of high-fidelity surface reconstruction by integrating SVRecon.<n>Our method achieves strong reconstruction accuracy while having consistently speedy convergence.
  arXiv  Detail & Related papers  (2025-11-21T16:32:01Z)
• Gesplat: Robust Pose-Free 3D Reconstruction via Geometry-Guided Gaussian Splatting [21.952325954391508]
  We introduce Gesplat, a 3DGS-based framework that enables robust novel view synthesis and geometrically consistent reconstruction from unposed sparse images.<n>Our approach achieves more robust performance on both forward-facing and large-scale complex datasets compared to other pose-free methods.
  arXiv  Detail & Related papers  (2025-10-11T08:13:46Z)
• Exploring Representation-Aligned Latent Space for Better Generation [86.45670422239317]
  We introduce ReaLS, which integrates semantic priors to improve generation performance.<n>We show that fundamental DiT and SiT trained on ReaLS can achieve a 15% improvement in FID metric.<n>The enhanced semantic latent space enables more perceptual downstream tasks, such as segmentation and depth estimation.
  arXiv  Detail & Related papers  (2025-02-01T07:42:12Z)
• MonoGSDF: Exploring Monocular Geometric Cues for Gaussian Splatting-Guided Implicit Surface Reconstruction [86.87464903285208]
  We introduce MonoGSDF, a novel method that couples primitives with a neural Signed Distance Field (SDF) for high-quality reconstruction.<n>To handle arbitrary-scale scenes, we propose a scaling strategy for robust generalization.<n>Experiments on real-world datasets outperforms prior methods while maintaining efficiency.
  arXiv  Detail & Related papers  (2024-11-25T20:07:07Z)
• Few-shot Learning as Cluster-induced Voronoi Diagrams: A Geometric Approach [12.382578792491747]
  Cluster-induced Voronoi Diagram (CIVD) improves the accuracy and robustness of few-shot learning. Our CIVD-based workflow enables us to achieve new state-of-the-art results on mini-ImageNet, CUB, and tiered-ImagenNet datasets.
  arXiv  Detail & Related papers  (2022-02-05T02:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.