RPBG: Towards Robust Neural Point-based Graphics in the Wild

• URL: http://arxiv.org/abs/2405.05663v2
• Date: Thu, 11 Jul 2024 03:17:28 GMT
• Title: RPBG: Towards Robust Neural Point-based Graphics in the Wild
• Authors: Qingtian Zhu, Zizhuang Wei, Zhongtian Zheng, Yifan Zhan, Zhuyu Yao, Jiawang Zhang, Kejian Wu, Yinqiang Zheng,
• Abstract summary: We propose a new point-based neural re-rendering method, known as Point-based Graphics (RPBG) RPBG stably outperforms the baseline by a large margin, and exhibits its great robustness over state-of-the-art NeRF-based variants.
• Score: 23.497693378880303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Point-based representations have recently gained popularity in novel view synthesis, for their unique advantages, e.g., intuitive geometric representation, simple manipulation, and faster convergence. However, based on our observation, these point-based neural re-rendering methods are only expected to perform well under ideal conditions and suffer from noisy, patchy points and unbounded scenes, which are challenging to handle but defacto common in real applications. To this end, we revisit one such influential method, known as Neural Point-based Graphics (NPBG), as our baseline, and propose Robust Point-based Graphics (RPBG). We in-depth analyze the factors that prevent NPBG from achieving satisfactory renderings on generic datasets, and accordingly reform the pipeline to make it more robust to varying datasets in-the-wild. Inspired by the practices in image restoration, we greatly enhance the neural renderer to enable the attention-based correction of point visibility and the inpainting of incomplete rasterization, with only acceptable overheads. We also seek for a simple and lightweight alternative for environment modeling and an iterative method to alleviate the problem of poor geometry. By thorough evaluation on a wide range of datasets with different shooting conditions and camera trajectories, RPBG stably outperforms the baseline by a large margin, and exhibits its great robustness over state-of-the-art NeRF-based variants. Code available at https://github.com/QT-Zhu/RPBG.

Related papers

• SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization [16.460851701725392]
  We present a novel approach that optimize radiance fields with scene graphs to mitigate the influence of outlier poses. Our method incorporates an adaptive inlier-outlier confidence estimation scheme based on scene graphs. We also introduce an effective intersection-over-union (IoU) loss to optimize the camera pose and surface geometry.
  arXiv  Detail & Related papers  (2024-07-17T15:50:17Z)
• SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance [106.0057551634008]
  FreeNeRF attempts to overcome this limitation by integrating implicit geometry regularization. New study introduces a novel feature matching based sparse geometry regularization module. module excels in pinpointing high-frequency keypoints, thereby safeguarding the integrity of fine details.
  arXiv  Detail & Related papers  (2024-04-01T08:37:57Z)
• High-Fidelity SLAM Using Gaussian Splatting with Rendering-Guided Densification and Regularized Optimization [8.845446246585215]
  We propose a dense RGBD SLAM system based on 3D Gaussian Splatting that provides metrically accurate pose tracking and visually realistic reconstruction. Compared to recent neural and concurrently developed gaussian splatting RGBD SLAM baselines, our method achieves state-of-the-art results on the synthetic dataset Replica and competitive results on the real-world dataset TUM.
  arXiv  Detail & Related papers  (2024-03-19T08:19:53Z)
• Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation [7.203073346844801]
  This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF) We propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering. Our approach explicitly models visibilities by geometric priors and augments them with neural features.
  arXiv  Detail & Related papers  (2024-01-25T17:58:51Z)
• Chasing Fairness in Graphs: A GNN Architecture Perspective [73.43111851492593]
  We propose textsfFair textsfMessage textsfPassing (FMP) designed within a unified optimization framework for graph neural networks (GNNs) In FMP, the aggregation is first adopted to utilize neighbors' information and then the bias mitigation step explicitly pushes demographic group node presentation centers together. Experiments on node classification tasks demonstrate that the proposed FMP outperforms several baselines in terms of fairness and accuracy on three real-world datasets.
  arXiv  Detail & Related papers  (2023-12-19T18:00:15Z)
• PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
  We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation. On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points. On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
  arXiv  Detail & Related papers  (2023-12-17T08:30:00Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detection [52.355018626115346]
  Building object detectors that are robust to domain shifts is critical for real-world applications. We propose to use Relative Gradient Norm as a way to measure the vulnerability of a backbone to feature distortion. We present recipes to boost OOD robustness for both types of backbones.
  arXiv  Detail & Related papers  (2023-03-26T14:50:43Z)
• NPBG++: Accelerating Neural Point-Based Graphics [14.366073496519139]
  NPBG++ is a novel view synthesis (NVS) task that achieves high rendering realism with low scene fitting time. Our method efficiently leverages the multiview observations and the point cloud of a static scene to predict a neural descriptor for each point. In our comparisons, the proposed system outperforms previous NVS approaches in terms of fitting and rendering runtimes while producing images of similar quality.
  arXiv  Detail & Related papers  (2022-03-24T19:59:39Z)
• Cross-Scale Internal Graph Neural Network for Image Super-Resolution [147.77050877373674]
  Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. For single image super-resolution (SISR), most existing deep non-local methods only exploit similar patches within the same scale of the low-resolution (LR) input image. This is achieved using a novel cross-scale internal graph neural network (IGNN)
  arXiv  Detail & Related papers  (2020-06-30T10:48:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.