Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement

• URL: http://arxiv.org/abs/2604.09473v1
• Date: Fri, 10 Apr 2026 16:31:12 GMT
• Title: Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
• Abstract summary: We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces.<n>We present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy.<n>We propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data.
• Score: 27.56981802996559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.

Related papers

• OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation [75.70865997182294]
  Human-Object Interaction Video Generation (HOIVG) aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose.<n>We present OmniShow, an end-to-end framework tailored for this practical yet challenging task.
  arXiv  Detail & Related papers  (2026-04-13T17:59:12Z)
• SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations [88.8747004592363]
  SceneScribe-1M is a new large-scale, multi-temporal video dataset.<n>It comprises one million in-the-wild videos, each meticulously annotated with detailed textual descriptions, precise parameters, dense depth maps, and consistent 3D point tracks.<n>We demonstrate the versatility and value of SceneScribe-1M by establishing benchmarks across a wide array of downstream tasks, including monocular depth estimation, scene reconstruction dynamic point tracking, as well as generative tasks such as text-to-video synthesis, with or without camera control.
  arXiv  Detail & Related papers  (2026-04-09T08:59:33Z)
• MultiEgo: A Multi-View Egocentric Video Dataset for 4D Scene Reconstruction [23.428989479526336]
  We present MultiEgo, the first multi-view egocentric dataset for 4D dynamic scene reconstruction.<n>The dataset comprises five canonical social interaction scenes: meetings, performances, and a presentation.<n>Experiment validation demonstrates the practical utility and effectiveness of our dataset for free-viewpoint video (FVV) applications.
  arXiv  Detail & Related papers  (2025-12-12T05:54:19Z)
• DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling [67.95038177144554]
  We introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video.<n>We employ vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic captions.<n> DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos.
  arXiv  Detail & Related papers  (2025-12-02T18:24:27Z)
• Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound [5.591620304505415]
  This work presents the first formal framework for Audio-Visual World Models (AVWM)<n>It formulates multimodal environment simulation as a partially observable decision process with audio-visual observations, fine-grained actions, and task rewards.<n>We propose an Audio-Visual Conditional Transformer with a novel modality expert architecture that balances visual and auditory learning.
  arXiv  Detail & Related papers  (2025-11-30T13:11:56Z)
• ImViD: Immersive Volumetric Videos for Enhanced VR Engagement [34.450247091615395]
  Next frontier in VR/AR technologies lies in immersive volumetric videos with complete scene capture, large 6-DoF interaction space, multi-modal feedback, and high resolution & frame-rate contents.<n>We introduce ImViD, a multi-view, multi-modal dataset featuring complete space-oriented data capture and various indoor/outdoor scenarios.<n>Our capture rig supports multi-view video-audio capture while on the move, significantly enhancing the completeness, flexibility, and efficiency of data capture.
  arXiv  Detail & Related papers  (2025-03-18T15:42:22Z)
• AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation [62.682428307810525]
  We introduce AVS-Mamba, a selective state space model to address the audio-visual segmentation task.<n>Our framework incorporates two key components for video understanding and cross-modal learning.<n>Our approach achieves new state-of-the-art results on the AVSBench-object and AVS-semantic datasets.
  arXiv  Detail & Related papers  (2025-01-14T03:20:20Z)
• EasyVolcap: Accelerating Neural Volumetric Video Research [69.59671164891725]
  Volumetric video is a technology that digitally records dynamic events such as artistic performances, sporting events, and remote conversations. EasyVolcap is a Python & Pytorch library for unifying the process of multi-view data processing, 4D scene reconstruction, and efficient dynamic volumetric video rendering.
  arXiv  Detail & Related papers  (2023-12-11T17:59:46Z)
• DynamicStereo: Consistent Dynamic Depth from Stereo Videos [91.1804971397608]
  We propose DynamicStereo to estimate disparity for stereo videos. The network learns to pool information from neighboring frames to improve the temporal consistency of its predictions. We also introduce Dynamic Replica, a new benchmark dataset containing synthetic videos of people and animals in scanned environments.
  arXiv  Detail & Related papers  (2023-05-03T17:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.