Sekai: A Video Dataset towards World Exploration

• URL: http://arxiv.org/abs/2506.15675v2
• Date: Fri, 20 Jun 2025 09:03:18 GMT
• Title: Sekai: A Video Dataset towards World Exploration
• Abstract summary: Sekai (meaning world'' in Japanese) is a high-quality first-person view worldwide video dataset with rich annotations for world exploration.<n>It consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from over 100 countries and regions across 750 cities.
• Score: 53.151247175736636
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video generation techniques have made remarkable progress, promising to be the foundation of interactive world exploration. However, existing video generation datasets are not well-suited for world exploration training as they suffer from some limitations: limited locations, short duration, static scenes, and a lack of annotations about exploration and the world. In this paper, we introduce Sekai (meaning ``world'' in Japanese), a high-quality first-person view worldwide video dataset with rich annotations for world exploration. It consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from over 100 countries and regions across 750 cities. We develop an efficient and effective toolbox to collect, pre-process and annotate videos with location, scene, weather, crowd density, captions, and camera trajectories. Experiments demonstrate the quality of the dataset. And, we use a subset to train an interactive video world exploration model, named YUME (meaning ``dream'' in Japanese). We believe Sekai will benefit the area of video generation and world exploration, and motivate valuable applications. The project page is https://lixsp11.github.io/sekai-project/.

Related papers

• Sekai2: From World Exploration to Interactive World Modeling [22.57939715452233]
  Sekai2 is a multi-source real-world video dataset that carries the world-exploration footage of Sekai toward interactive world modeling.<n>The release contains 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries or regions.<n>Every clip includes a released camera trajectory and hierarchical annotations disentangling subject motion, environment dynamics, static scene content, and camera behavior, resulting in 649,597 temporally grounded segments.
  arXiv  Detail & Related papers  (2026-08-10T11:23:01Z)
• Yume: An Interactive World Generation Model [38.818537395166835]
  Yume aims to use images, text, or videos to create an interactive, realistic, and dynamic world.<n>Method creates a dynamic world from an input image and allows exploration of the world using keyboard actions.
  arXiv  Detail & Related papers  (2025-07-23T17:57:09Z)
• GenWorld: Towards Detecting AI-generated Real-world Simulation Videos [79.98542193919957]
  GenWorld is a large-scale, high-quality, and real-world simulation dataset for AI-generated video detection.<n>We propose a model, SpannDetector, to leverage multi-view consistency as a strong criterion for real-world AI-generated video detection.
  arXiv  Detail & Related papers  (2025-06-12T17:59:33Z)
• WorldExplorer: Towards Generating Fully Navigable 3D Scenes [49.21733308718443]
  WorldExplorer builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints.<n>We generate multiple videos along short, pre-defined trajectories, that explore the scene in depth.<n>Our novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results.
  arXiv  Detail & Related papers  (2025-06-02T15:41:31Z)
• Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation [2.4240014793575138]
  As of this writing, OpenAI's Sora, the current state-of-the-art system, is still limited to producing videos that are up to one minute in length.<n>In this survey, we examine the current landscape of long video generation, covering techniques like GANs and diffusion models, video generation strategies, large-scale training datasets, quality metrics for evaluating long videos, and future research areas to address the limitations of the existing video generation capabilities.
  arXiv  Detail & Related papers  (2024-12-24T21:24:41Z)
• From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos [71.22810401256234]
  Three-dimensional (3D) understanding of objects and scenes play a key role in humans' ability to interact with the world.<n>Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects.<n>We introduce 360-1M, a 360 video dataset, and a process for efficiently finding corresponding frames from diverse viewpoints at scale.
  arXiv  Detail & Related papers  (2024-12-10T18:59:44Z)
• CityGuessr: City-Level Video Geo-Localization on a Global Scale [54.371452373726584]
  We propose a novel problem of worldwide video geolocalization with the objective of hierarchically predicting the correct city, state/province, country, and continent, given a video. No large scale video datasets that have extensive worldwide coverage exist, to train models for solving this problem. We introduce a new dataset, CityGuessr68k comprising of 68,269 videos from 166 cities all over the world.
  arXiv  Detail & Related papers  (2024-11-10T03:20:00Z)
• WonderWorld: Interactive 3D Scene Generation from a Single Image [38.83667648993784]
  We present WonderWorld, a novel framework for interactive 3D scene generation.<n>WonderWorld generates connected and diverse 3D scenes in less than 10 seconds on a single A6000 GPU.
  arXiv  Detail & Related papers  (2024-06-13T17:59:10Z)
• A Semi-Self-Supervised Approach for Dense-Pattern Video Object Segmentation [6.092973123903838]
  We propose a semi-self-supervised approach for dense-VOS (DVOS) using a diffusion-based method through multi-task learning.<n>We evaluate our DVOS method for wheat head segmentation from a diverse set of videos, spanning from Boot-stage to Wheat-mature and Harvest-ready.<n>While our method was evaluated on wheat head segmentation, it can be extended to other crops and domains, such as crowd analysis or microscopic image analysis.
  arXiv  Detail & Related papers  (2024-06-07T17:58:36Z)
• Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception? [57.77643186237265]
  We present Multiview Aerial Visual RECognition or MAVREC, a video dataset where we record synchronized scenes from different perspectives. MAVREC consists of around 2.5 hours of industry-standard 2.7K resolution video sequences, more than 0.5 million frames, and 1.1 million annotated bounding boxes. This makes MAVREC the largest ground and aerial-view dataset, and the fourth largest among all drone-based datasets.
  arXiv  Detail & Related papers  (2023-12-07T18:59:14Z)
• PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking [90.29143475328506]
  We introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. We animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos.
  arXiv  Detail & Related papers  (2023-07-27T17:58:11Z)
• FSVVD: A Dataset of Full Scene Volumetric Video [2.9151420469958533]
  In this paper, we focus on the current most widely used data format, point cloud, and for the first time release a full-scene volumetric video dataset. Comprehensive dataset description and analysis are conducted, with potential usage of this dataset.
  arXiv  Detail & Related papers  (2023-03-07T02:31:08Z)
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer [66.56167074658697]
  We present a method that builds on 3D-VQGAN and transformers to generate videos with thousands of frames. Our evaluation shows that our model trained on 16-frame video clips can generate diverse, coherent, and high-quality long videos. We also showcase conditional extensions of our approach for generating meaningful long videos by incorporating temporal information with text and audio.
  arXiv  Detail & Related papers  (2022-04-07T17:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.