4D Visualization of Dynamic Events from Unconstrained Multi-View Videos
- URL: http://arxiv.org/abs/2005.13532v1
- Date: Wed, 27 May 2020 17:57:19 GMT
- Title: 4D Visualization of Dynamic Events from Unconstrained Multi-View Videos
- Authors: Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, Srinivasa
Narasimhan
- Abstract summary: We present a data-driven approach for 4D space-time visualization of dynamic events from videos captured by hand-held multiple cameras.
Key to our approach is the use of self-supervised neural networks specific to the scene to compose static and dynamic aspects of an event.
This model allows us to create virtual cameras that facilitate: (1) freezing the time and exploring views; (2) freezing a view and moving through time; and (3) simultaneously changing both time and view.
- Score: 77.48430951972928
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a data-driven approach for 4D space-time visualization of dynamic
events from videos captured by hand-held multiple cameras. Key to our approach
is the use of self-supervised neural networks specific to the scene to compose
static and dynamic aspects of an event. Though captured from discrete
viewpoints, this model enables us to move around the space-time of the event
continuously. This model allows us to create virtual cameras that facilitate:
(1) freezing the time and exploring views; (2) freezing a view and moving
through time; and (3) simultaneously changing both time and view. We can also
edit the videos and reveal occluded objects for a given view if it is visible
in any of the other views. We validate our approach on challenging in-the-wild
events captured using up to 15 mobile cameras.
Related papers
- Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis [43.02778060969546]
We propose a controllable monocular dynamic view synthesis pipeline.
Our model does not require depth as input, and does not explicitly model 3D scene geometry.
We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.
arXiv Detail & Related papers (2024-05-23T17:59:52Z) - Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras [65.54875149514274]
We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel.
At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose.
It handles actors in wide clothing, and reproduces even fine-scale dynamic detail.
arXiv Detail & Related papers (2023-12-12T16:45:52Z) - Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from
a Single Image [59.18564636990079]
We study the problem of synthesizing a long-term dynamic video from only a single image.
Existing methods either hallucinate inconsistent perpetual views or struggle with long camera trajectories.
We present Make-It-4D, a novel method that can generate a consistent long-term dynamic video from a single image.
arXiv Detail & Related papers (2023-08-20T12:53:50Z) - Decoupling Dynamic Monocular Videos for Dynamic View Synthesis [50.93409250217699]
We tackle the challenge of dynamic view synthesis from dynamic monocular videos in an unsupervised fashion.
Specifically, we decouple the motion of the dynamic objects into object motion and camera motion, respectively regularized by proposed unsupervised surface consistency and patch-based multi-view constraints.
arXiv Detail & Related papers (2023-04-04T11:25:44Z) - A Portable Multiscopic Camera for Novel View and Time Synthesis in
Dynamic Scenes [42.00094186447837]
We present a portable multiscopic camera system with a dedicated model for novel view and time synthesis in dynamic scenes.
Our goal is to render high-quality images for a dynamic scene from any viewpoint at any time using our portable multiscopic camera.
arXiv Detail & Related papers (2022-08-30T17:53:17Z) - Playable Environments: Video Manipulation in Space and Time [98.0621309257937]
We present Playable Environments - a new representation for interactive video generation and manipulation in space and time.
With a single image at inference time, our novel framework allows the user to move objects in 3D while generating a video by providing a sequence of desired actions.
Our method builds an environment state for each frame, which can be manipulated by our proposed action module and decoded back to the image space with volumetric rendering.
arXiv Detail & Related papers (2022-03-03T18:51:05Z) - Learning to Deblur and Rotate Motion-Blurred Faces [43.673660541417995]
We train a neural network to reconstruct a 3D video representation from a single image and the corresponding face gaze.
We then provide a camera viewpoint relative to the estimated gaze and the blurry image as input to an encoder-decoder network to generate a video of sharp frames with a novel camera viewpoint.
arXiv Detail & Related papers (2021-12-14T17:51:19Z) - Recognizing Actions in Videos from Unseen Viewpoints [80.6338404141284]
We show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in training data.
We introduce a new dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.
arXiv Detail & Related papers (2021-03-30T17:17:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.