CE-NPBG: Connectivity Enhanced Neural Point-Based Graphics for Novel View Synthesis in Autonomous Driving Scenes

• URL: http://arxiv.org/abs/2504.19557v1
• Date: Mon, 28 Apr 2025 08:02:02 GMT
• Title: CE-NPBG: Connectivity Enhanced Neural Point-Based Graphics for Novel View Synthesis in Autonomous Driving Scenes
• Authors: Mohammad Altillawi, Fengyi Shen, Liudi Yang, Sai Manoj Prakhya, Ziyuan Liu,
• Abstract summary: We present CE-NPBG, a new approach for novel view synthesis (NVS) in large-scale autonomous driving scenes.<n>Our method is a neural point-based technique that leverages two modalities: posed images (cameras) and synchronized raw 3D point clouds (LiDAR)<n>By leveraging this connectivity, our method significantly improves rendering quality and enhances run-time and scalability.
• Score: 5.719388462440881
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current point-based approaches encounter limitations in scalability and rendering quality when using large 3D point cloud maps because using them directly for novel view synthesis (NVS) leads to degraded visualizations. We identify the primary issue behind these low-quality renderings as a visibility mismatch between geometry and appearance, stemming from using these two modalities together. To address this problem, we present CE-NPBG, a new approach for novel view synthesis (NVS) in large-scale autonomous driving scenes. Our method is a neural point-based technique that leverages two modalities: posed images (cameras) and synchronized raw 3D point clouds (LiDAR). We first employ a connectivity relationship graph between appearance and geometry, which retrieves points from a large 3D point cloud map observed from the current camera perspective and uses them for rendering. By leveraging this connectivity, our method significantly improves rendering quality and enhances run-time and scalability by using only a small subset of points from the large 3D point cloud map. Our approach associates neural descriptors with the points and uses them to synthesize views. To enhance the encoding of these descriptors and elevate rendering quality, we propose a joint adversarial and point rasterization training. During training, we pair an image-synthesizer network with a multi-resolution discriminator. At inference, we decouple them and use the image-synthesizer to generate novel views. We also integrate our proposal into the recent 3D Gaussian Splatting work to highlight its benefits for improved rendering and scalability.

Related papers

• Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis [2.612019169899311]
  We present PointmapDiffusion, a novel framework for single-image novel view synthesis.<n>Our method is the first to leverage pointmaps as a conditioning signal, capturing prior to the reference images to guide the diffusion process.<n>Experiments on diverse real-world datasets demonstrate that PointmapDiffusion achieves high-quality, multi-view consistent results.
  arXiv  Detail & Related papers  (2025-01-06T10:48:31Z)
• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures.<n>Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression [1.2974519529978974]
  This paper introduces a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF) generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis.
  arXiv  Detail & Related papers  (2024-03-15T13:40:37Z)
• Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration [107.61458720202984]
  This paper introduces a novel self-supervised learning framework for enhancing 3D perception in autonomous driving scenes.<n>We propose the learnable transformation alignment to bridge the domain gap between image and point cloud data.<n>We establish dense 2D-3D correspondences to estimate the rigid pose.
  arXiv  Detail & Related papers  (2024-01-23T02:41:06Z)
• VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering [4.542331789204584]
  We present a novel neural-rendering-based approach to detect and fix deficiencies in novel view synthesis. We show that our approach can improve the quality of a point cloud obtained by structure from motion. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively.
  arXiv  Detail & Related papers  (2023-11-08T12:23:57Z)
• Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields [63.21420081888606]
  Recent Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. We present Point2Pix as a novel point to link the 3D sparse point clouds with 2D dense image pixels.
  arXiv  Detail & Related papers  (2023-03-29T06:26:55Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Learning A Locally Unified 3D Point Cloud for View Synthesis [45.757280092357355]
  We propose a new deep learning-based view synthesis paradigm that learns a locally unified 3D point cloud from source views. Experimental results on three benchmark datasets demonstrate that our method can improve the average PSNR by more than 4 dB.
  arXiv  Detail & Related papers  (2022-09-12T04:07:34Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• View Synthesis with Sculpted Neural Points [64.40344086212279]
  Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency. We propose a new approach that performs view synthesis using point clouds. It is the first point-based method to achieve better visual quality than NeRF while being more than 100x faster in rendering speed.
  arXiv  Detail & Related papers  (2022-05-12T03:54:35Z)
• IDEA-Net: Dynamic 3D Point Cloud Interpolation via Deep Embedding Alignment [58.8330387551499]
  We formulate the problem as estimation of point-wise trajectories (i.e., smooth curves) We propose IDEA-Net, an end-to-end deep learning framework, which disentangles the problem under the assistance of the explicitly learned temporal consistency. We demonstrate the effectiveness of our method on various point cloud sequences and observe large improvement over state-of-the-art methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2022-03-22T10:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.