T-SVG: Text-Driven Stereoscopic Video Generation

• URL: http://arxiv.org/abs/2412.09323v1
• Date: Thu, 12 Dec 2024 14:48:46 GMT
• Title: T-SVG: Text-Driven Stereoscopic Video Generation
• Authors: Qiao Jin, Xiaodong Chen, Wu Liu, Tao Mei, Yongdong Zhang,
• Abstract summary: This paper introduces the Text-driven Stereoscopic Video Generation (T-SVG) system. It streamlines video generation by using text prompts to create reference videos. These videos are transformed into 3D point cloud sequences, which are rendered from two perspectives with subtle parallax differences.
• Score: 87.62286959918566
• License:
• Abstract: The advent of stereoscopic videos has opened new horizons in multimedia, particularly in extended reality (XR) and virtual reality (VR) applications, where immersive content captivates audiences across various platforms. Despite its growing popularity, producing stereoscopic videos remains challenging due to the technical complexities involved in generating stereo parallax. This refers to the positional differences of objects viewed from two distinct perspectives and is crucial for creating depth perception. This complex process poses significant challenges for creators aiming to deliver convincing and engaging presentations. To address these challenges, this paper introduces the Text-driven Stereoscopic Video Generation (T-SVG) system. This innovative, model-agnostic, zero-shot approach streamlines video generation by using text prompts to create reference videos. These videos are transformed into 3D point cloud sequences, which are rendered from two perspectives with subtle parallax differences, achieving a natural stereoscopic effect. T-SVG represents a significant advancement in stereoscopic content creation by integrating state-of-the-art, training-free techniques in text-to-video generation, depth estimation, and video inpainting. Its flexible architecture ensures high efficiency and user-friendliness, allowing seamless updates with newer models without retraining. By simplifying the production pipeline, T-SVG makes stereoscopic video generation accessible to a broader audience, demonstrating its potential to revolutionize the field.

Related papers

• TexAVi: Generating Stereoscopic VR Video Clips from Text Descriptions [0.562479170374811]
  This paper proposes an approach to coalesce existing generative systems to form a stereoscopic virtual reality video from text. Our work highlights the exciting possibilities of using natural language-driven graphics in fields like virtual reality simulations.
  arXiv  Detail & Related papers  (2025-01-02T09:21:03Z)
• UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics [74.10447111842504]
  UniReal is a unified framework designed to address various image generation and editing tasks. Inspired by recent video generation models, we propose a unifying approach that treats image-level tasks as discontinuous video generation. Although designed for image-level tasks, we leverage videos as a scalable source for universal supervision.
  arXiv  Detail & Related papers  (2024-12-10T18:59:55Z)
• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks [25.39030226963548]
  We introduce the first application of a pretrained transformer-based video generative model for portrait animation. Our method is validated through experiments on benchmark and newly proposed wild datasets.
  arXiv  Detail & Related papers  (2024-12-01T08:54:30Z)
• Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model And Input View Curation [61.040832373015014]
  We propose Flex3D, a novel framework for generating high-quality 3D content from text, single images, or sparse view images. We employ a fine-tuned multi-view image diffusion model and a video diffusion model to generate a pool of candidate views, enabling a rich representation of the target 3D object. In the second stage, the curated views are fed into a Flexible Reconstruction Model (FlexRM), built upon a transformer architecture that can effectively process an arbitrary number of inputs.
  arXiv  Detail & Related papers  (2024-10-01T17:29:43Z)
• StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos [44.51044100125421]
  This paper presents a novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience. Our framework demonstrates significant improvements in 2D-to-3D video conversion, offering a practical solution for creating immersive content for 3D devices like Apple Vision Pro and 3D displays.
  arXiv  Detail & Related papers  (2024-09-11T17:52:07Z)
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis [63.169364481672915]
  We propose textbfViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames.
  arXiv  Detail & Related papers  (2024-09-03T16:53:19Z)
• One-Shot Learning Meets Depth Diffusion in Multi-Object Videos [0.0]
  This paper introduces a novel depth-conditioning approach that enables the generation of coherent and diverse videos from just a single text-video pair. Our method fine-tunes the pre-trained model to capture continuous motion by employing custom-designed spatial and temporal attention mechanisms. During inference, we use the DDIM inversion to provide structural guidance for video generation.
  arXiv  Detail & Related papers  (2024-08-29T16:58:10Z)
• SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix [60.48666051245761]
  We propose a pose-free and training-free approach for generating 3D stereoscopic videos. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth. We develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting.
  arXiv  Detail & Related papers  (2024-06-29T08:33:55Z)
• Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization [52.63845811751936]
  Video pre-training is challenging due to the modeling of its dynamics video. In this paper, we address such limitations in video pre-training with an efficient video decomposition. Our framework is both capable of comprehending and generating image and video content, as demonstrated by its performance across 13 multimodal benchmarks.
  arXiv  Detail & Related papers  (2024-02-05T16:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.