ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

• URL: http://arxiv.org/abs/2312.09095v2
• Date: Fri, 15 Dec 2023 02:03:30 GMT
• Title: ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field
• Authors: Zhangkai Ni, Peiqi Yang, Wenhan Yang, Hanli Wang, Lin Ma, Sam Kwong
• Abstract summary: Collaborative Neural Radiance Fields (ColNeRF) is designed to work with sparse input. ColNeRF is capable of capturing richer and more generalized scene representation. Our approach exhibits superiority in fine-tuning towards adapting to new scenes.
• Score: 89.54363625953044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) have demonstrated impressive potential in synthesizing novel views from dense input, however, their effectiveness is challenged when dealing with sparse input. Existing approaches that incorporate additional depth or semantic supervision can alleviate this issue to an extent. However, the process of supervision collection is not only costly but also potentially inaccurate, leading to poor performance and generalization ability in diverse scenarios. In our work, we introduce a novel model: the Collaborative Neural Radiance Fields (ColNeRF) designed to work with sparse input. The collaboration in ColNeRF includes both the cooperation between sparse input images and the cooperation between the output of the neural radiation field. Through this, we construct a novel collaborative module that aligns information from various views and meanwhile imposes self-supervised constraints to ensure multi-view consistency in both geometry and appearance. A Collaborative Cross-View Volume Integration module (CCVI) is proposed to capture complex occlusions and implicitly infer the spatial location of objects. Moreover, we introduce self-supervision of target rays projected in multiple directions to ensure geometric and color consistency in adjacent regions. Benefiting from the collaboration at the input and output ends, ColNeRF is capable of capturing richer and more generalized scene representation, thereby facilitating higher-quality results of the novel view synthesis. Extensive experiments demonstrate that ColNeRF outperforms state-of-the-art sparse input generalizable NeRF methods. Furthermore, our approach exhibits superiority in fine-tuning towards adapting to new scenes, achieving competitive performance compared to per-scene optimized NeRF-based methods while significantly reducing computational costs. Our code is available at: https://github.com/eezkni/ColNeRF.

Related papers

• SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance [106.0057551634008]
  FreeNeRF attempts to overcome this limitation by integrating implicit geometry regularization. New study introduces a novel feature matching based sparse geometry regularization module. module excels in pinpointing high-frequency keypoints, thereby safeguarding the integrity of fine details.
  arXiv  Detail & Related papers  (2024-04-01T08:37:57Z)
• CombiNeRF: A Combination of Regularization Techniques for Few-Shot Neural Radiance Field View Synthesis [1.374796982212312]
  Neural Radiance Fields (NeRFs) have shown impressive results for novel view synthesis when a sufficiently large amount of views are available. We propose CombiNeRF, a framework that synergically combines several regularization techniques, some of them novel, in order to unify the benefits of each. We show that CombiNeRF outperforms the state-of-the-art methods with few-shot settings in several publicly available datasets.
  arXiv  Detail & Related papers  (2024-03-21T13:59:00Z)
• CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency [18.101763989542828]
  We propose a simple yet effective method that explicitly builds depth-aware consistency across input views. Our key insight is that by forcing the same spatial points to be sampled repeatedly in different input views, we are able to strengthen the interactions between views. Although simple, extensive experiments demonstrate that our proposed method can achieve better synthesis quality over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-26T09:04:04Z)
• Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering [3.8200916793910973]
  Recent advances in Neural Radiance Fields (NeRF) have demonstrated significant potential for representing 3D scene appearances as implicit neural networks. However, the lengthy training and rendering process hinders the widespread adoption of this promising technique for real-time rendering applications. We present an effective adaptive multi-NeRF method designed to accelerate the neural rendering process for large scenes.
  arXiv  Detail & Related papers  (2023-10-03T08:34:49Z)
• Grid-guided Neural Radiance Fields for Large Urban Scenes [146.06368329445857]
  Recent approaches propose to geographically divide the scene and adopt multiple sub-NeRFs to model each region individually. An alternative solution is to use a feature grid representation, which is computationally efficient and can naturally scale to a large scene. We present a new framework that realizes high-fidelity rendering on large urban scenes while being computationally efficient.
  arXiv  Detail & Related papers  (2023-03-24T13:56:45Z)
• IntrinsicNGP: Intrinsic Coordinate based Hash Encoding for Human NeRF [27.660829835903424]
  We propose IntrinsicNGP, which can train from scratch and achieve high-fidelity results in few minutes with videos of a human performer. We introduce a continuous and optimizable intrinsic coordinate rather than the original explicit Euclidean coordinate in the hash encoding module of instant-NGP. With this novel intrinsic coordinate, IntrinsicNGP can aggregate inter-frame information for dynamic objects with the help of proxy geometry shapes.
  arXiv  Detail & Related papers  (2023-02-28T15:51:19Z)
• IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis [90.03590032170169]
  We present intrinsic neural radiance fields, dubbed IntrinsicNeRF, which introduce intrinsic decomposition into the NeRF-based neural rendering method. Our experiments and editing samples on both object-specific/room-scale scenes and synthetic/real-word data demonstrate that we can obtain consistent intrinsic decomposition results.
  arXiv  Detail & Related papers  (2022-10-02T22:45:11Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.