Sekai2: From World Exploration to Interactive World Modeling

• URL: http://arxiv.org/abs/2608.09449v2
• Date: Tue, 11 Aug 2026 09:05:36 GMT
• Title: Sekai2: From World Exploration to Interactive World Modeling
• Abstract summary: Sekai2 is a multi-source real-world video dataset that carries the world-exploration footage of Sekai toward interactive world modeling.<n>The release contains 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries or regions.<n>Every clip includes a released camera trajectory and hierarchical annotations disentangling subject motion, environment dynamics, static scene content, and camera behavior, resulting in 649,597 temporally grounded segments.
• Score: 22.57939715452233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video world models must capture how scenes evolve over time and across viewpoints. Training them for long-horizon generation and camera control therefore benefits from long videos paired with camera trajectories and temporally grounded semantics. Existing corpora rarely offer the three together: large-scale web video provides broad visual diversity but no trajectories or time-aligned text, while pose-annotated datasets are typically short-range or reconstruction-oriented. We introduce Sekai2, a multi-source real-world video dataset that carries the world-exploration footage of Sekai toward interactive world modeling. The release contains 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries or regions, and is deliberately weighted toward sustained observation: under a common 120-second decomposition, 43,594 segments reach the full two minutes and account for 51.4% of all footage. Every clip includes a released camera trajectory and hierarchical annotations disentangling subject motion, environment dynamics, static scene content, and camera behavior, resulting in 649,597 temporally grounded segments. Crucially, we further introduce 982 panoramic sequences captured along non-linear trajectories with loops and revisits. These revisits provide repeated observations of the same locations across time and viewpoints, offering essential supervision for learning persistent scene representations, long-term spatial memory, and geometrically consistent world models. Corpus-scale analyses demonstrate complete pose-and-caption coverage, broad geographic and semantic diversity, varied camera trajectories, and highly non-redundant temporal descriptions. Together, these properties make Sekai2 a scalable resource for long-horizon video generation, camera-controllable synthesis, and interactive world-model pre-training.

Related papers

• Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics [72.11325009788095]
  We introduce Kairos, a video dataset for video-language modeling with time-resolved annotations.<n>Kairos consists of long-duration videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal alignment.
  arXiv  Detail & Related papers  (2026-09-08T13:50:30Z)
• Wonder: Video World Model Done Better [70.28870858365214]
  We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration.<n>Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera.<n>We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues.
  arXiv  Detail & Related papers  (2026-07-28T17:45:25Z)
• SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations [88.8747004592363]
  SceneScribe-1M is a new large-scale, multi-temporal video dataset.<n>It comprises one million in-the-wild videos, each meticulously annotated with detailed textual descriptions, precise parameters, dense depth maps, and consistent 3D point tracks.<n>We demonstrate the versatility and value of SceneScribe-1M by establishing benchmarks across a wide array of downstream tasks, including monocular depth estimation, scene reconstruction dynamic point tracking, as well as generative tasks such as text-to-video synthesis, with or without camera control.
  arXiv  Detail & Related papers  (2026-04-09T08:59:33Z)
• OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation [42.159343032593014]
  We propose OmniRoam, a controllable panoramic video generation framework.<n>Our framework exploits the rich per-frame scene coverage and inherent long-term spatial and temporal consistency of panoramic representation.<n> Experiments show that our framework consistently outperforms state-of-the-art methods in terms of visual quality, controllability, and long-term scene consistency.
  arXiv  Detail & Related papers  (2026-03-31T17:59:33Z)
• WorldReel: 4D Video Generation with Consistent Geometry and Motion Modeling [63.37476802589492]
  We present WorldReel, a 4D video that is native-temporally consistent.<n>WorldReel produces frames together with 4D scene representations, including pointmaps, camera trajectory, and dense flow.<n>We believe that WorldReel brings video generation closer to 4D-consistent world modeling.
  arXiv  Detail & Related papers  (2025-12-08T18:54:12Z)
• SpatialVID: A Large-Scale Video Dataset with Spatial Annotations [58.01259302233675]
  SpatialVID is a dataset of in-the-wild videos with diverse scenes, camera movements and dense 3D annotations such as per-frame camera poses, depth, and motion instructions.<n>We collect more than 21,000 hours of raw video, and process them into 2.7 million clips through a hierarchical filtering pipeline.<n>A subsequent annotation pipeline enriches these clips with detailed spatial and semantic information, including camera poses, depth maps, dynamic masks, structured captions, and serialized motion instructions.
  arXiv  Detail & Related papers  (2025-09-11T17:59:31Z)
• Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation [66.95956271144982]
  We present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image.<n>Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames.
  arXiv  Detail & Related papers  (2025-06-04T17:59:04Z)
• DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal Consistent Video Generation [20.512252799625685]
  S-temporal consistency is a critical research topic in video generation.<n>This paper introduces integral-temporal consistency, considering the synergy between plot progression and camera techniques.<n>We develop and train the DropletVideo model, which excels in preserving dynamic-temporal coherence during video generation.
  arXiv  Detail & Related papers  (2025-03-08T04:37:38Z)
• PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking [90.29143475328506]
  We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. We animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos.
  arXiv  Detail & Related papers  (2023-07-27T17:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.