Stereo Magnification with Multi-Layer Images
- URL: http://arxiv.org/abs/2201.05023v1
- Date: Thu, 13 Jan 2022 15:19:46 GMT
- Title: Stereo Magnification with Multi-Layer Images
- Authors: Taras Khakhulin, Denis Korzhenkov, Pavel Solovev, Gleb Sterkin,
Timotei Ardelean, Victor Lempitsky
- Abstract summary: We introduce a new view synthesis approach based on multiple semi-transparent layers with scene-adapted geometry.
Our approach infers such representations from stereo pairs in two stages.
In experiments, we demonstrate the advantage of the proposed approach over the use of regularly-spaced layers.
- Score: 7.20447989181373
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Representing scenes with multiple semi-transparent colored layers has been a
popular and successful choice for real-time novel view synthesis. Existing
approaches infer colors and transparency values over regularly-spaced layers of
planar or spherical shape. In this work, we introduce a new view synthesis
approach based on multiple semi-transparent layers with scene-adapted geometry.
Our approach infers such representations from stereo pairs in two stages. The
first stage infers the geometry of a small number of data-adaptive layers from
a given pair of views. The second stage infers the color and the transparency
values for these layers producing the final representation for novel view
synthesis. Importantly, both stages are connected through a differentiable
renderer and are trained in an end-to-end manner. In the experiments, we
demonstrate the advantage of the proposed approach over the use of
regularly-spaced layers with no adaptation to scene geometry. Despite being
orders of magnitude faster during rendering, our approach also outperforms a
recently proposed IBRNet system based on implicit geometry representation. See
results at https://samsunglabs.github.io/StereoLayers .
Related papers
- Volumetric Surfaces: Representing Fuzzy Geometries with Multiple Meshes [59.17785932398617]
High-quality real-time view synthesis methods are based on volume rendering, splatting, or surface rendering.
We present a novel representation for real-time view where the number of sampling locations is small and bounded.
We show that our method can represent challenging fuzzy objects while achieving higher frame rates than volume-based and splatting-based methods on low-end and mobile devices.
arXiv Detail & Related papers (2024-09-04T07:18:26Z) - Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information.
We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering.
Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
arXiv Detail & Related papers (2023-04-20T04:05:22Z) - BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature
Fusion for Deep Exemplar-based Video Colorization [70.14893481468525]
We present an effective BiSTNet to explore colors of reference exemplars and utilize them to help video colorization.
We first establish the semantic correspondence between each frame and the reference exemplars in deep feature space to explore color information from reference exemplars.
We develop a mixed expert block to extract semantic information for modeling the object boundaries of frames so that the semantic image prior can better guide the colorization process.
arXiv Detail & Related papers (2022-12-05T13:47:15Z) - WALDO: Future Video Synthesis using Object Layer Decomposition and
Parametric Flow Prediction [82.79642869586587]
WALDO is a novel approach to the prediction of future video frames from past ones.
Individual images are decomposed into multiple layers combining object masks and a small set of control points.
The layer structure is shared across all frames in each video to build dense inter-frame connections.
arXiv Detail & Related papers (2022-11-25T18:59:46Z) - Self-improving Multiplane-to-layer Images for Novel View Synthesis [3.9901365062418312]
We present a new method for lightweight novel-view synthesis that generalizes to an arbitrary forward-facing scene.
We start by representing the scene with a set of fronto-parallel semitransparent planes and afterward convert them to deformable layers in an end-to-end manner.
Our method does not require fine-tuning when a new scene is processed and can handle an arbitrary number of views without restrictions.
arXiv Detail & Related papers (2022-10-04T13:27:14Z) - Single-View View Synthesis in the Wild with Learned Adaptive Multiplane
Images [15.614631883233898]
Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations.
We propose a new method based on the multiplane image (MPI) representation.
The experiments on both synthetic and real datasets demonstrate that our trained model works remarkably well and achieves state-of-the-art results.
arXiv Detail & Related papers (2022-05-24T02:57:16Z) - TRANSPR: Transparency Ray-Accumulating Neural 3D Scene Point Renderer [6.320273914694594]
We propose and evaluate a neural point-based graphics method that can model semi-transparent scene parts.
We show that novel views of semi-transparent point cloud scenes can be generated after training with our approach.
arXiv Detail & Related papers (2020-09-06T21:19:18Z) - Free View Synthesis [100.86844680362196]
We present a method for novel view synthesis from input images that are freely distributed around a scene.
Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.
arXiv Detail & Related papers (2020-08-12T18:16:08Z) - Single-View View Synthesis with Multiplane Images [64.46556656209769]
We apply deep learning to generate multiplane images given two or more input images at known viewpoints.
Our method learns to predict a multiplane image directly from a single image input.
It additionally generates reasonable depth maps and fills in content behind the edges of foreground objects in background layers.
arXiv Detail & Related papers (2020-04-23T17:59:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.