Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesis

• URL: http://arxiv.org/abs/2505.00135v1
• Date: Wed, 30 Apr 2025 19:06:09 GMT
• Title: Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesis
• Authors: Michal Geyer, Omer Tov, Linyi Jin, Richard Tucker, Inbar Mosseri, Tali Dekel, Noah Snavely,
• Abstract summary: Despite advances in video synthesis, creating 3D videos remains challenging due to the relative scarcity of 3D video data.<n>We propose a simple approach for transforming a text-to-video generator into a video-to-stereo generator.<n>Our framework automatically produces the video frames from a shifted viewpoint, enabling a compelling 3D effect.
• Score: 45.64047250474718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rising popularity of immersive visual experiences has increased interest in stereoscopic 3D video generation. Despite significant advances in video synthesis, creating 3D videos remains challenging due to the relative scarcity of 3D video data. We propose a simple approach for transforming a text-to-video generator into a video-to-stereo generator. Given an input video, our framework automatically produces the video frames from a shifted viewpoint, enabling a compelling 3D effect. Prior and concurrent approaches for this task typically operate in multiple phases, first estimating video disparity or depth, then warping the video accordingly to produce a second view, and finally inpainting the disoccluded regions. This approach inherently fails when the scene involves specular surfaces or transparent objects. In such cases, single-layer disparity estimation is insufficient, resulting in artifacts and incorrect pixel shifts during warping. Our work bypasses these restrictions by directly synthesizing the new viewpoint, avoiding any intermediate steps. This is achieved by leveraging a pre-trained video model's priors on geometry, object materials, optics, and semantics, without relying on external geometry models or manually disentangling geometry from the synthesis process. We demonstrate the advantages of our approach in complex, real-world scenarios featuring diverse object materials and compositions. See videos on https://video-eye2eye.github.io

Related papers

• VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step [13.168559963356952]
  VideoScene aims to distill the video diffusion model to generate 3D scenes in one step.<n>VideoScene achieves faster and superior 3D scene generation results than previous video diffusion models.
  arXiv  Detail & Related papers  (2025-04-02T17:59:21Z)
• LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis [80.2461057573121]
  In this work, we augment the interaction with a new dimension, i.e., the depth dimension, such that users are allowed to assign a relative depth for each point on the trajectory.<n>We propose a pioneering method for 3D trajectory control in image-to-video by abstracting object masks into a few cluster points.<n>Experiments validate the effectiveness of our approach, dubbed LeviTor, in precisely manipulating the object movements when producing photo-realistic videos from static images.
  arXiv  Detail & Related papers  (2024-12-19T18:59:56Z)
• OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos [7.616167860385134]
  It has long been challenging to recover the underlying dynamic 3D scene representations from a monocular RGB video. We introduce a new framework, called OSN, to learn all plausible 3D scene configurations that match the input video. Our method demonstrates a clear advantage in learning fine-grained 3D scene geometry.
  arXiv  Detail & Related papers  (2024-07-08T05:03:46Z)
• SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix [60.48666051245761]
  We propose a pose-free and training-free approach for generating 3D stereoscopic videos. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth. We develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting.
  arXiv  Detail & Related papers  (2024-06-29T08:33:55Z)
• Splatter a Video: Video Gaussian Representation for Versatile Processing [48.9887736125712]
  Video representation is crucial for various down-stream tasks, such as tracking,depth prediction,segmentation,view synthesis,and editing. We introduce a novel explicit 3D representation-video Gaussian representation -- that embeds a video into 3D Gaussians. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation.
  arXiv  Detail & Related papers  (2024-06-19T22:20:03Z)
• Lester: rotoscope animation through video object segmentation and tracking [0.0]
  Lester is a novel method to automatically synthetise retro-style 2D animations from videos. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT. Results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances.
  arXiv  Detail & Related papers  (2024-02-15T11:15:54Z)
• GenDeF: Learning Generative Deformation Field for Video Generation [89.49567113452396]
  We propose to render a video by warping one static image with a generative deformation field (GenDeF) Such a pipeline enjoys three appealing advantages.
  arXiv  Detail & Related papers  (2023-12-07T18:59:41Z)
• WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction [82.79642869586587]
  WALDO is a novel approach to the prediction of future video frames from past ones. Individual images are decomposed into multiple layers combining object masks and a small set of control points. The layer structure is shared across all frames in each video to build dense inter-frame connections.
  arXiv  Detail & Related papers  (2022-11-25T18:59:46Z)
• Copy Motion From One to Another: Fake Motion Video Generation [53.676020148034034]
  A compelling application of artificial intelligence is to generate a video of a target person performing arbitrary desired motion. Current methods typically employ GANs with a L2 loss to assess the authenticity of the generated videos. We propose a theoretically motivated Gromov-Wasserstein loss that facilitates learning the mapping from a pose to a foreground image. Our method is able to generate realistic target person videos, faithfully copying complex motions from a source person.
  arXiv  Detail & Related papers  (2022-05-03T08:45:22Z)
• Unsupervised object-centric video generation and decomposition in 3D [36.08064849807464]
  We propose to model a video as the view seen while moving through a scene with multiple 3D objects and a 3D background. Our model is trained from monocular videos without any supervision, yet learns to generate coherent 3D scenes containing several moving objects.
  arXiv  Detail & Related papers  (2020-07-07T18:01:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.