CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis

• URL: http://arxiv.org/abs/2501.14533v1
• Date: Fri, 24 Jan 2025 14:40:39 GMT
• Title: CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis
• Authors: Konstantinos Georgiadis, Mehmet Kerim Yucel, Albert Saa-Garriga,
• Abstract summary: Single-view novel view synthesis (NVS) is a notorious problem due to its ill-posed nature, and often requires large, computationally expensive approaches to produce tangible results. We propose CheapNVS: a fully end-to-end approach for narrow baseline single-view NVS based on a novel, efficient multiple encoder/decoder design trained in a multi-stage fashion.
• Score: 2.4578723416255754
• License:
• Abstract: Single-view novel view synthesis (NVS) is a notorious problem due to its ill-posed nature, and often requires large, computationally expensive approaches to produce tangible results. In this paper, we propose CheapNVS: a fully end-to-end approach for narrow baseline single-view NVS based on a novel, efficient multiple encoder/decoder design trained in a multi-stage fashion. CheapNVS first approximates the laborious 3D image warping with lightweight learnable modules that are conditioned on the camera pose embeddings of the target view, and then performs inpainting on the occluded regions in parallel to achieve significant performance gains. Once trained on a subset of Open Images dataset, CheapNVS outperforms the state-of-the-art despite being 10 times faster and consuming 6% less memory. Furthermore, CheapNVS runs comfortably in real-time on mobile devices, reaching over 30 FPS on a Samsung Tab 9+.

Related papers

• NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images [50.36605863731669]
  NVComposer is a novel approach that eliminates the need for explicit external alignment. NVComposer achieves state-of-the-art performance in generative multi-view NVS tasks. Our approach shows substantial improvements in synthesis quality as the number of unposed input views increases.
  arXiv  Detail & Related papers  (2024-12-04T17:58:03Z)
• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures. Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes [50.534213038479926]
  FreeSplat is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. We propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views.
  arXiv  Detail & Related papers  (2024-05-28T08:40:14Z)
• Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion [62.37374499337897]
  We present Dual3D, a novel text-to-3D generation framework. It generates high-quality 3D assets from texts in only $1$ minute.
  arXiv  Detail & Related papers  (2024-05-16T07:50:02Z)
• fMPI: Fast Novel View Synthesis in the Wild with Layered Scene Representations [9.75588035624177]
  We propose two novel input processing paradigms for novel view synthesis (NVS) methods. Our approach identifies and mitigates the two most time-consuming aspects of traditional pipelines. We demonstrate that our proposed paradigms enable the design of an NVS method that achieves state-of-the-art on public benchmarks.
  arXiv  Detail & Related papers  (2023-12-26T16:24:08Z)
• Novel View Synthesis with View-Dependent Effects from a Single Image [35.85973300177698]
  We first consider view-dependent effects into single image-based novel view synthesis (NVS) problems. We propose to exploit the camera motion priors in NVS to model view-dependent appearance or effects (VDE) as the negative disparity in the scene. We present extensive experiment results and show that our proposed method can learn NVS with VDEs, outperforming the SOTA single-view NVS methods on the RealEstate10k and MannequinChallenge datasets.
  arXiv  Detail & Related papers  (2023-12-13T11:29:47Z)
• Novel View Synthesis from a Single RGBD Image for Indoor Scenes [4.292698270662031]
  We propose an approach for synthesizing novel view images from a single RGBD (Red Green Blue-Depth) input. In our method, we convert an RGBD image into a point cloud and render it from a different viewpoint, then formulate the NVS task into an image translation problem.
  arXiv  Detail & Related papers  (2023-11-02T08:34:07Z)
• TOSS:High-quality Text-guided Novel View Synthesis from a Single Image [36.90122394242858]
  We present TOSS, which introduces text to the task of novel view synthesis (NVS) from just a single RGB image. To address this limitation, TOSS uses text as high-level semantic information to constrain the NVS solution space.
  arXiv  Detail & Related papers  (2023-10-16T17:59:09Z)
• Stitched ViTs are Flexible Vision Backbones [51.441023711924835]
  We are inspired by stitchable neural networks (SN-Net) to produce a single model that covers richworks by stitching pretrained model families. We introduce SN-Netv2, a systematically improved model stitching framework to facilitate downstream task adaptation. SN-Netv2 demonstrates superior performance over SN-Netv1 on downstream dense predictions and shows strong ability as a flexible vision backbone.
  arXiv  Detail & Related papers  (2023-06-30T22:05:34Z)
• S-VolSDF: Sparse Multi-View Stereo Regularization of Neural Implicit Surfaces [75.30792581941789]
  Neural rendering of implicit surfaces performs well in 3D vision applications. When only sparse input images are available, output quality drops significantly due to the shape-radiance ambiguity problem. We propose to regularize neural rendering optimization with an MVS solution.
  arXiv  Detail & Related papers  (2023-03-30T21:10:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.