Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective

• URL: http://arxiv.org/abs/2604.14025v1
• Date: Wed, 15 Apr 2026 16:07:18 GMT
• Title: Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
• Abstract summary: Generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years.<n>Existing feed-forward approaches share similar high-level architectural patterns.<n>We propose a novel taxonomy centered on model design strategies that are agnostic to the output format.
• Score: 91.23306722968509
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reconstructing 3D representations from 2D inputs is a fundamental task in computer vision and graphics, serving as a cornerstone for understanding and interacting with the physical world. While traditional methods achieve high fidelity, they are limited by slow per-scene optimization or category-specific training, which hinders their practical deployment and scalability. Hence, generalizable feed-forward 3D reconstruction has witnessed rapid development in recent years. By learning a model that maps images directly to 3D representations in a single forward pass, these methods enable efficient reconstruction and robust cross-scene generalization. Our survey is motivated by a critical observation: despite the diverse geometric output representations, ranging from implicit fields to explicit primitives, existing feed-forward approaches share similar high-level architectural patterns, such as image feature extraction backbones, multi-view information fusion mechanisms, and geometry-aware design principles. Consequently, we abstract away from these representation differences and instead focus on model design, proposing a novel taxonomy centered on model design strategies that are agnostic to the output format. Our proposed taxonomy organizes the research directions into five key problems that drive recent research development: feature enhancement, geometry awareness, model efficiency, augmentation strategies and temporal-aware models. To support this taxonomy with empirical grounding and standardized evaluation, we further comprehensively review related benchmarks and datasets, and extensively discuss and categorize real-world applications based on feed-forward 3D models. Finally, we outline future directions to address open challenges such as scalability, evaluation standards, and world modeling.

Related papers

• A Cookbook of 3D Vision: Data, Learning Paradigms, and Application [22.14736170010237]
  3D vision has rapidly evolved, driven by increasingly diverse data representations, learning paradigms, and modeling strategies.<n>This work provides a data-centric taxonomy of 3D vision that connects geometric representations, datasets, learning frameworks, and applications within a single conceptual map.
  arXiv  Detail & Related papers  (2026-06-02T23:41:52Z)
• GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion [29.999238067855245]
  We propose GTA, a novel image-to-3D world generation method following a Geometry-Then-Appearance paradigm.<n>Specifically, given a single input image, GTA adopts a two-stage framework with two dedicated video diffusion models.<n>Extensive experiments have demonstrated that our proposed method consistently outperforms existing approaches in terms of fidelity, visual quality, and geometric accuracy.
  arXiv  Detail & Related papers  (2026-05-13T03:43:02Z)
• Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors [61.34273238077091]
  We present a novel method for generating orbital videos from a single image of an object.<n>Our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-14T05:35:46Z)
• 3D and 4D World Modeling: A Survey [104.20852751473392]
  World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit.<n>We introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches.<n>We discuss practical applications, identify open challenges, and highlight promising research directions.
  arXiv  Detail & Related papers  (2025-09-04T17:59:58Z)
• Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey [171.72616707259306]
  3D reconstruction and view synthesis are foundational problems in computer vision, graphics, and immersive technologies such as augmented reality (AR), virtual reality (VR), and digital twins.<n>Recent advances in feed-forward approaches, driven by deep learning, have revolutionized this field by enabling fast and generalizable 3D reconstruction and view synthesis.
  arXiv  Detail & Related papers  (2025-07-19T06:13:25Z)
• Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation [96.1872246747684]
  Depth estimation is a fundamental task in 3D computer vision, crucial for applications such as 3D reconstruction, free-viewpoint rendering, robotics, autonomous driving, and AR/VR technologies.<n>Traditional methods relying on hardware sensors like LiDAR are often limited by high costs, low resolution, and environmental sensitivity, limiting their applicability in real-world scenarios.<n>Recent advances in vision-based methods offer a promising alternative, yet they face challenges in generalization and stability due to either the low-capacity model architectures or the reliance on domain-specific and small-scale datasets.
  arXiv  Detail & Related papers  (2025-07-15T17:59:59Z)
• Review of Feed-forward 3D Reconstruction: From DUSt3R to VGGT [10.984522161856955]
  3D reconstruction is a cornerstone technology for numerous applications, including augmented/virtual reality, autonomous driving, and robotics.<n>Deep learning has catalyzed a paradigm shift in 3D reconstruction.<n>New models employ a unified deep network to jointly infer camera poses and dense geometry directly from an Unconstrained set of images in a single forward pass.
  arXiv  Detail & Related papers  (2025-07-11T09:41:54Z)
• Recent Advance in 3D Object and Scene Generation: A Survey [14.673302810271219]
  This survey aims to provide readers with a structured understanding of state-of-the-art 3D generation technologies.<n>We focus on three dominant paradigms: layout-guided compositional synthesis, 2D prior-based scene generation, and rule-driven modeling.
  arXiv  Detail & Related papers  (2025-04-16T03:22:06Z)
• Aether: Geometric-Aware Unified World Modeling [49.33579903601599]
  Aether is a unified framework that enables geometry-aware reasoning in world models.<n>Our framework achieves zero-shot generalization in both action following and reconstruction tasks.<n>We hope our work inspires the community to explore new frontiers in physically-reasonable world modeling.
  arXiv  Detail & Related papers  (2025-03-24T17:59:51Z)
• Scalable Scene Modeling from Perspective Imaging: Physics-based Appearance and Geometry Inference [3.2229099973277076]
  dissertation presents a fraction of contributions that advances 3D scene modeling to its state of the art. In contrast to the prevailing deep learning methods, as a core contribution, this thesis aims to develop algorithms that follow first principles.
  arXiv  Detail & Related papers  (2024-04-01T17:09:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.