OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

• URL: http://arxiv.org/abs/2509.12201v2
• Date: Wed, 24 Sep 2025 13:15:38 GMT
• Title: OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
• Abstract summary: We introduce OmniWorld, a large-scale, multi-domain, multi-modal dataset specifically designed for 4D world modeling.<n>Compared with existing synthetic datasets, OmniWorld-Game provides richer modality coverage, larger scale, and more realistic dynamic interactions.<n>We establish a challenging benchmark that exposes the limitations of current state-of-the-art (SOTA) approaches in modeling complex 4D environments.
• Score: 86.12242953301121
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of 4D world modeling - aiming to jointly capture spatial geometry and temporal dynamics - has witnessed remarkable progress in recent years, driven by advances in large-scale generative models and multimodal learning. However, the development of truly general 4D world models remains fundamentally constrained by the availability of high-quality data. Existing datasets and benchmarks often lack the dynamic complexity, multi-domain diversity, and spatial-temporal annotations required to support key tasks such as 4D geometric reconstruction, future prediction, and camera-control video generation. To address this gap, we introduce OmniWorld, a large-scale, multi-domain, multi-modal dataset specifically designed for 4D world modeling. OmniWorld consists of a newly collected OmniWorld-Game dataset and several curated public datasets spanning diverse domains. Compared with existing synthetic datasets, OmniWorld-Game provides richer modality coverage, larger scale, and more realistic dynamic interactions. Based on this dataset, we establish a challenging benchmark that exposes the limitations of current state-of-the-art (SOTA) approaches in modeling complex 4D environments. Moreover, fine-tuning existing SOTA methods on OmniWorld leads to significant performance gains across 4D reconstruction and video generation tasks, strongly validating OmniWorld as a powerful resource for training and evaluation. We envision OmniWorld as a catalyst for accelerating the development of general-purpose 4D world models, ultimately advancing machines' holistic understanding of the physical world.

Related papers

• Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation [7.87892459538874]
  We introduce WANDA: learning open-World mobile mANipulation from one demonstration via a synthetic DAta engine.<n>We apply Corrective State Expansion to increase the robot and object state diversity at different stages of mobile manipulation.<n>Experiments show that policies trained with WANDA achieve long-horizon robustness, broad spatial generalization and cross-environment generalization.
  arXiv  Detail & Related papers  (2026-07-14T18:04:58Z)
• Embody4D: A Generalist 4D World Model for Embodied AI [13.488596291534243]
  Embody4D is a dedicated video-to-video world model for embodied scenarios.<n>It is capable of arbitrary novel views from a monocular video.<n>It achieves state-of-the-art performance, serving as a robust world model.
  arXiv  Detail & Related papers  (2026-05-03T09:39:44Z)
• MotionScape: A Large-Scale Real-World Highly Dynamic UAV Video Dataset for World Models [6.680381992984212]
  We present MotionScape, a large-scale real-world UAVview video dataset with highly dynamic motion for world modeling.<n>MotionScape contains over 30 hours of 4K UAV-view videos, totaling more than 4.5M frames.<n>This novel dataset features semantically and geometrically aligned training samples, where diverse real-world UAV videos are tightly coupled with accurate 6-DoF camera trajectories and fine-grained natural language descriptions.
  arXiv  Detail & Related papers  (2026-04-09T08:59:40Z)
• Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models [39.648000265543445]
  Video-based world models have emerged along two dominant paradigms: video generation and 3D reconstruction.<n>We argue that the future of world modeling lies in 4D generation, which jointly models spatial structure and temporal evolution.<n>We propose Omni--WorldBench, a benchmark specifically designed to evaluate the interactive response capabilities of world models in 4D settings.
  arXiv  Detail & Related papers  (2026-03-23T17:10:29Z)
• Beyond Pixel Histories: World Models with Persistent 3D State [50.4601060508243]
  PERSIST is a new paradigm of world model which simulates the evolution of a latent 3D scene.<n>We show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods.
  arXiv  Detail & Related papers  (2026-03-03T19:58:31Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling [67.95038177144554]
  We introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video.<n>We employ vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic captions.<n> DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos.
  arXiv  Detail & Related papers  (2025-12-02T18:24:27Z)
• 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models [29.06964332825464]
  World Generation Models are emerging as a cornerstone of next-generation multimodal intelligence systems.<n>World Models aim to construct realistic, dynamic, and physically consistent 3D/4D worlds from images, videos, or text.<n>We introduce the 4DWorldBench, which measures models across four key dimensions: Perceptual Quality, Condition-4D Alignment, Physical Realism, and 4D Consistency.
  arXiv  Detail & Related papers  (2025-11-25T02:05:35Z)
• SPATIALGEN: Layout-guided 3D Indoor Scene Generation [37.30623176278608]
  We present SpatialGen, a novel multi-view multi-modal diffusion model that generates realistic and semantically consistent 3D indoor scenes.<n>Given a 3D layout and a reference image, our model synthesizes appearance (color image), geometry (scene coordinate map), and semantic (semantic segmentation map) from arbitrary viewpoints.<n>We are open-sourcing our data and models to empower the community and advance the field of indoor scene understanding and generation.
  arXiv  Detail & Related papers  (2025-09-18T14:12:32Z)
• LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation [35.4193352348583]
  We propose a simple yet effective 3D world generation framework that streamlines the industrial production pipeline of 3D environments.<n>LatticeWorld creates large-scale 3D interactive worlds with dynamic agents, featuring competitive multi-agent interaction.<n>LatticeWorld achieves over a $90times$ increase in industrial production efficiency.
  arXiv  Detail & Related papers  (2025-09-05T17:22:33Z)
• PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model [23.768571323272152]
  PartRM is a novel 4D reconstruction framework that simultaneously models appearance, geometry, and part-level motion from multi-view images of a static object.<n>We introduce the PartDrag-4D dataset, providing multi-view observations of part-level dynamics across over 20,000 states.<n> Experimental results show that PartRM establishes a new state-of-the-art in part-level motion learning and can be applied in manipulation tasks in robotics.
  arXiv  Detail & Related papers  (2025-03-25T17:59:58Z)
• GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control [122.65089441381741]
  We present GEM, a Generalizable Ego-vision Multimodal world model.<n>It predicts future frames using a reference frame, sparse features, human poses, and ego-trajectories.<n>Our dataset is comprised of 4000+ hours of multimodal data across domains like autonomous driving, egocentric human activities, and drone flights.
  arXiv  Detail & Related papers  (2024-12-15T14:21:19Z)
• DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model [65.43473733967038]
  We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. Our dataset features video clips with a complete set of driving maneuvers, diverse multi-agent interplay, and rich open-world driving knowledge.
  arXiv  Detail & Related papers  (2024-10-14T17:19:23Z)
• OmniRe: Omni Urban Scene Reconstruction [78.99262488964423]
  We introduce OmniRe, a comprehensive system for creating high-fidelity digital twins of dynamic real-world scenes from on-device logs.<n>Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors.
  arXiv  Detail & Related papers  (2024-08-29T17:56:33Z)
• 3D-VLA: A 3D Vision-Language-Action Generative World Model [68.0388311799959]
  Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. We propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments.
  arXiv  Detail & Related papers  (2024-03-14T17:58:41Z)
• LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment [59.320414108383055]
  We present LiveHPS, a novel single-LiDAR-based approach for scene-level human pose and shape estimation. We propose a huge human motion dataset, named FreeMotion, which is collected in various scenarios with diverse human poses.
  arXiv  Detail & Related papers  (2024-02-27T03:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.