VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs

• URL: http://arxiv.org/abs/2304.13386v2
• Date: Fri, 2 Jun 2023 05:39:27 GMT
• Title: VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs
• Authors: Jiakai Sun, Zhanjie Zhang, Jiafu Chen, Guangyuan Li, Boyan Ji, Lei Zhao, Wei Xing, Huaizhong Lin
• Abstract summary: VGOS is an approach for fast (3-5 minutes) radiance field reconstruction from sparse inputs (3-10 views) We introduce an incremental voxel training strategy, which prevents overfitting by suppressing the optimization of peripheral voxels. Experiments demonstrate that VGOS achieves state-of-the-art performance for sparse inputs with super-fast convergence.
• Score: 9.374561178958404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) has shown great success in novel view synthesis due to its state-of-the-art quality and flexibility. However, NeRF requires dense input views (tens to hundreds) and a long training time (hours to days) for a single scene to generate high-fidelity images. Although using the voxel grids to represent the radiance field can significantly accelerate the optimization process, we observe that for sparse inputs, the voxel grids are more prone to overfitting to the training views and will have holes and floaters, which leads to artifacts. In this paper, we propose VGOS, an approach for fast (3-5 minutes) radiance field reconstruction from sparse inputs (3-10 views) to address these issues. To improve the performance of voxel-based radiance field in sparse input scenarios, we propose two methods: (a) We introduce an incremental voxel training strategy, which prevents overfitting by suppressing the optimization of peripheral voxels in the early stage of reconstruction. (b) We use several regularization techniques to smooth the voxels, which avoids degenerate solutions. Experiments demonstrate that VGOS achieves state-of-the-art performance for sparse inputs with super-fast convergence. Code will be available at https://github.com/SJoJoK/VGOS.

Related papers

• Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering [37.48219196092378]
  We propose an efficient radiance field rendering algorithm that incorporates a radianceization process on sparse voxels without neural networks or 3D Gaussians. We adaptively fit sparse voxels to different levels of detail within scenes, faithfully reproducing details while achieving high rendering frame rates. Our method improves the previous neural-free voxel grid representation by over 4db PSNR and more than 10x rendering FPS speedup.
  arXiv  Detail & Related papers  (2024-12-05T18:59:11Z)
• DGTR: Distributed Gaussian Turbo-Reconstruction for Sparse-View Vast Scenes [81.56206845824572]
  Novel-view synthesis (NVS) approaches play a critical role in vast scene reconstruction. Few-shot methods often struggle with poor reconstruction quality in vast environments. This paper presents DGTR, a novel distributed framework for efficient Gaussian reconstruction for sparse-view vast scenes.
  arXiv  Detail & Related papers  (2024-11-19T07:51:44Z)
• MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields [73.49548565633123]
  Radiance fields represented by 3D Gaussians excel at synthesizing novel views, offering both high training efficiency and fast rendering. Existing methods often incorporate depth priors from dense estimation networks but overlook the inherent multi-view consistency in input images. We propose a view framework based on 3D Gaussian Splatting, named MCGS, enabling scene reconstruction from sparse input views.
  arXiv  Detail & Related papers  (2024-10-15T08:39:05Z)
• Spatial Annealing for Efficient Few-shot Neural Rendering [73.49548565633123]
  We introduce an accurate and efficient few-shot neural rendering method named textbfSpatial textbfAnnealing regularized textbfNeRF (textbfSANeRF) By adding merely one line of code, SANeRF delivers superior rendering quality and much faster reconstruction speed compared to current few-shot neural rendering methods.
  arXiv  Detail & Related papers  (2024-06-12T02:48:52Z)
• CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance Fields from Sparse Inputs [65.80187860906115]
  We propose a novel approach to improve NeRF's performance with sparse inputs. We first adopt a voxel-based ray sampling strategy to ensure that the sampled rays intersect with a certain voxel in 3D space. We then randomly sample additional points within the voxel and apply a Transformer to infer the properties of other points on each ray, which are then incorporated into the volume rendering.
  arXiv  Detail & Related papers  (2024-03-25T15:56:17Z)
• VoxNeRF: Bridging Voxel Representation and Neural Radiance Fields for Enhanced Indoor View Synthesis [73.50359502037232]
  VoxNeRF is a novel approach to enhance the quality and efficiency of neural indoor reconstruction and novel view synthesis. We propose an efficient voxel-guided sampling technique that allocates computational resources to selectively the most relevant segments of rays. Our approach is validated with extensive experiments on ScanNet and ScanNet++.
  arXiv  Detail & Related papers  (2023-11-09T11:32:49Z)
• RealLiFe: Real-Time Light Field Reconstruction via Hierarchical Sparse Gradient Descent [23.4659443904092]
  EffLiFe is a novel light field optimization method that produces high-quality light fields from sparse view images in real time. Our method achieves comparable visual quality while being 100x faster on average than state-of-the-art offline methods.
  arXiv  Detail & Related papers  (2023-07-06T14:31:01Z)
• DARF: Depth-Aware Generalizable Neural Radiance Field [51.29437249009986]
  We propose the Depth-Aware Generalizable Neural Radiance Field (DARF) with a Depth-Aware Dynamic Sampling (DADS) strategy. Our framework infers the unseen scenes on both pixel level and geometry level with only a few input images. Compared with state-of-the-art generalizable NeRF methods, DARF reduces samples by 50%, while improving rendering quality and depth estimation.
  arXiv  Detail & Related papers  (2022-12-05T14:00:59Z)
• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction [42.3230709881297]
  We present a super-fast convergence approach to reconstructing the per-scene radiance field from a set of images. Our approach achieves NeRF-comparable quality and converges rapidly from scratch in less than 15 minutes with a single GPU.
  arXiv  Detail & Related papers  (2021-11-22T14:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.