Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models

• URL: http://arxiv.org/abs/2607.19120v1
• Date: Tue, 21 Jul 2026 14:06:04 GMT
• Title: Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models
• Abstract summary: Generative models for 3D scenes must rely on strong geometric priors to produce coherent outputs from sparse inputs.<n>We bridge these two paradigms by performing flow matching directly in VGGT's latent space.<n>On RealEstate10K, ScanNet++ and ETH3D, our method achieves strong performance against recent scene generation baselines.
• Score: 27.12023248183448
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Geometric foundation models, such as the Visual Geometry Grounded Transformer (VGGT), provide strong 3D priors from unposed images. However, such models operate purely in a feed-forward, deterministic regime, \ie~they cannot generate plausible geometry beyond what the input views directly support. Generative models for 3D scenes, on the other hand, must rely on strong geometric priors to produce coherent outputs from sparse inputs. We bridge these two paradigms by performing flow matching directly in VGGT's latent space, leveraging its learned 3D priors without committing to any explicit downstream representation such as Gaussians, meshes, or video-VAE latents. This requires respecting the latent geometry: VGGT tokens occupy a product of high-dimensional hyperspheres on which standard Euclidean flow matching fails. We address this with a Riemannian Flow Matching framework defined on a product manifold of four hyperspheres, aligned with VGGT's multi-scale encoder, which keeps generated tokens on the valid data manifold required by the frozen decoding heads. On RealEstate10K, ScanNet++ and ETH3D, our method achieves strong performance against recent scene generation baselines in both per-view appearance and aggregated 3D geometry, establishing latent-space flow matching on geometric foundation models as a viable paradigm for 3D generation. The project page can be found $\href{https://lisaweijler.github.io/geometry-grounded-rfm/}{\text{here}}$.

Related papers

• Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models [12.061080782435013]
  Recent Vision Foundation Models (VFMs) predict depth, camera pose, and pointmap in a single forward pass without per-scene optimization.<n> enforcing explicit multi-view geometric consistency, e.g., through bundle adjustment, is computationally costly and is thus not imposed during VFM pretraining.<n>We propose Self-Geometry, a plug-and-play test-time adaptation pipeline that directly imposes explicit multi-view geometric constraints using 2D pixel correspondences as pseudo ground-truth.
  arXiv  Detail & Related papers  (2026-08-11T09:28:50Z)
• FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation [72.38727895659405]
  We introduce FLAT, which decodes triangle splats directly from video diffusion latents.<n>On standard benchmarks, FLAT achieves significantly better geometric accuracy compared to state-of-the-art feedforward baselines.<n>We show that a lightweight test-time refinement step converts the predicted triangle soup into a fully opaque, game-engine-ready representation.
  arXiv  Detail & Related papers  (2026-06-23T17:53:41Z)
• GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models [70.61152292499737]
  We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm.<n>Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision.<n>We propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation.
  arXiv  Detail & Related papers  (2026-03-17T12:43:48Z)
• GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentation [57.8059956428009]
  Recent attempts to transfer features from 2D Vision-Language Models to 3D semantic segmentation expose a persistent trade-off.<n>We propose GeoPurify that applies a small Student Affinity Network to 2D VLM-generated 3D point features using geometric priors distilled from a 3D self-supervised teacher model.<n>Benefiting from latent geometric information and the learned affinity network, GeoPurify effectively mitigates the trade-off and achieves superior data efficiency.
  arXiv  Detail & Related papers  (2025-10-02T16:37:56Z)
• UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generation [98.40254523605581]
  UniLat3D is a unified framework that encodes geometry and appearance in a single latent space.<n>Our key contribution is a geometry-appearance Unified VAE, which compresses high-resolution sparse features into a compact latent representation.<n>UniLat3D produces high-quality 3D assets in seconds from a single image.
  arXiv  Detail & Related papers  (2025-09-29T17:21:23Z)
• Dens3R: A Foundation Model for 3D Geometry Prediction [44.13431776180547]
  Dens3R is a 3D foundation model designed for joint geometric dense prediction.<n>By integrating image-pair matching features with intrinsic invariance modeling, Dens3R accurately regresses multiple geometric quantities.
  arXiv  Detail & Related papers  (2025-07-22T07:22:30Z)
• Dragen3D: Multiview Geometry Consistent 3D Gaussian Generation with Drag-Based Control [2.5031284037888395]
  Dragen3D is a novel approach that achieves geometrically consistent and controllable 3D generation.<n>We introduce the Anchor-Gaussian Variational Autoencoder (Anchor-GS VAE), which encodes a point cloud and a single image into anchor latents and decode these latents into 3DGS.<n>To the best of our knowledge, we are the first to achieve geometrically controllable 3D Gaussian generation and editing.
  arXiv  Detail & Related papers  (2025-02-23T07:19:03Z)
• Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation [27.43973967994717]
  MT3D is a text-to-3D generative model that leverages a high-fidelity 3D object to overcome viewpoint bias.<n>By incorporating geometric details from a 3D asset, MT3D enables the creation of diverse and geometrically consistent objects.
  arXiv  Detail & Related papers  (2024-08-12T06:25:44Z)
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation [65.33726478659304]
  We introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-06-21T17:49:31Z)
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis [90.26556260531707]
  DMTet is a conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology.
  arXiv  Detail & Related papers  (2021-11-08T05:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.