From 2D to 3D Cognition: A Brief Survey of General World Models

• URL: http://arxiv.org/abs/2506.20134v1
• Date: Wed, 25 Jun 2025 05:05:09 GMT
• Title: From 2D to 3D Cognition: A Brief Survey of General World Models
• Authors: Ningwei Xie, Zizi Tian, Lei Yang, Xiao-Ping Zhang, Meng Guo, Jie Li,
• Abstract summary: 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments.<n>Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models.<n>This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition.
• Score: 16.121071388463694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.

Related papers

• Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey [154.50661618628433]
  3D reconstruction and view synthesis are foundational problems in computer vision, graphics, and immersive technologies such as augmented reality (AR), virtual reality (VR), and digital twins.<n>Recent advances in feed-forward approaches, driven by deep learning, have revolutionized this field by enabling fast and generalizable 3D reconstruction and view synthesis.
  arXiv  Detail & Related papers  (2025-07-19T06:13:25Z)
• Simulating the Real World: A Unified Survey of Multimodal Generative Models [48.35284571052435]
  We present a unified survey for multimodal generative models that investigate the progression of data dimensionality in real-world simulation.<n>To the best of our knowledge, this is the first attempt to systematically unify the study of 2D, video, 3D and 4D generation within a single framework.
  arXiv  Detail & Related papers  (2025-03-06T17:31:43Z)
• Generative AI Framework for 3D Object Generation in Augmented Reality [0.0]
  This thesis integrates state-of-the-art generative AI models for real-time creation of 3D objects in augmented reality (AR) environments.<n>The framework demonstrates applications across industries such as gaming, education, retail, and interior design.<n>A significant contribution is democratizing 3D model creation, making advanced AI tools accessible to a broader audience.
  arXiv  Detail & Related papers  (2025-02-21T17:01:48Z)
• Diffusion Models in 3D Vision: A Survey [18.805222552728225]
  3D vision has become a crucial field within computer vision, powering a range of applications such as autonomous driving, robotics, augmented reality, and medical imaging.<n>We review the state-of-the-art methods that use diffusion models for 3D visual tasks, including but not limited to 3D object generation, shape completion, point-cloud reconstruction, and scene construction.<n>We discuss potential solutions, including improving computational efficiency, enhancing multimodal fusion, and exploring the use of large-scale pretraining for better generalization across 3D tasks.
  arXiv  Detail & Related papers  (2024-10-07T04:12:23Z)
• 3D-VLA: A 3D Vision-Language-Action Generative World Model [68.0388311799959]
  Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. We propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments.
  arXiv  Detail & Related papers  (2024-03-14T17:58:41Z)
• Progress and Prospects in 3D Generative AI: A Technical Overview including 3D human [51.58094069317723]
  This paper aims to provide a comprehensive overview and summary of the relevant papers published mostly during the latter half year of 2023. It will begin by discussing the AI generated object models in 3D, followed by the generated 3D human models, and finally, the generated 3D human motions, culminating in a conclusive summary and a vision for the future.
  arXiv  Detail & Related papers  (2024-01-05T03:41:38Z)
• Uni3D: Exploring Unified 3D Representation at Scale [66.26710717073372]
  We present Uni3D, a 3D foundation model to explore the unified 3D representation at scale. Uni3D uses a 2D ViT end-to-end pretrained to align the 3D point cloud features with the image-text aligned features. We show that the strong Uni3D representation also enables applications such as 3D painting and retrieval in the wild.
  arXiv  Detail & Related papers  (2023-10-10T16:49:21Z)
• Deep Generative Models on 3D Representations: A Survey [81.73385191402419]
  Generative models aim to learn the distribution of observed data by generating new instances. Recently, researchers started to shift focus from 2D to 3D space. representing 3D data poses significantly greater challenges.
  arXiv  Detail & Related papers  (2022-10-27T17:59:50Z)
• 3D Neural Scene Representations for Visuomotor Control [78.79583457239836]
  We learn models for dynamic 3D scenes purely from 2D visual observations. A dynamics model, constructed over the learned representation space, enables visuomotor control for challenging manipulation tasks.
  arXiv  Detail & Related papers  (2021-07-08T17:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.