rpcPRF: Generalizable MPI Neural Radiance Field for Satellite Camera
- URL: http://arxiv.org/abs/2310.07179v1
- Date: Wed, 11 Oct 2023 04:05:11 GMT
- Title: rpcPRF: Generalizable MPI Neural Radiance Field for Satellite Camera
- Authors: Tongtong Zhang, Yuanxiang Li
- Abstract summary: This paper presents rpcPRF, a Multiplane Images (MPI) based Planar neural Radiance Field for Rational Polynomial Camera (RPC)
We propose to use reprojection supervision to induce the predicted MPI to learn the correct geometry between the 3D coordinates and the images.
We remove the stringent requirement of dense depth supervision from deep multiview-stereo-based methods by introducing rendering techniques of radiance fields.
- Score: 0.76146285961466
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Novel view synthesis of satellite images holds a wide range of practical
applications. While recent advances in the Neural Radiance Field have
predominantly targeted pin-hole cameras, and models for satellite cameras often
demand sufficient input views. This paper presents rpcPRF, a Multiplane Images
(MPI) based Planar neural Radiance Field for Rational Polynomial Camera (RPC).
Unlike coordinate-based neural radiance fields in need of sufficient views of
one scene, our model is applicable to single or few inputs and performs well on
images from unseen scenes. To enable generalization across scenes, we propose
to use reprojection supervision to induce the predicted MPI to learn the
correct geometry between the 3D coordinates and the images. Moreover, we remove
the stringent requirement of dense depth supervision from deep
multiview-stereo-based methods by introducing rendering techniques of radiance
fields. rpcPRF combines the superiority of implicit representations and the
advantages of the RPC model, to capture the continuous altitude space while
learning the 3D structure. Given an RGB image and its corresponding RPC, the
end-to-end model learns to synthesize the novel view with a new RPC and
reconstruct the altitude of the scene. When multiple views are provided as
inputs, rpcPRF exerts extra supervision provided by the extra views. On the TLC
dataset from ZY-3, and the SatMVS3D dataset with urban scenes from WV-3, rpcPRF
outperforms state-of-the-art nerf-based methods by a significant margin in
terms of image fidelity, reconstruction accuracy, and efficiency, for both
single-view and multiview task.
Related papers
- psPRF:Pansharpening Planar Neural Radiance Field for Generalized 3D Reconstruction Satellite Imagery [0.6445605125467574]
Most current NeRF variants for satellites are designed for one specific scene and fall short of generalization to new geometry.
This paper introduces psPRF, a Planar Neural Radiance Field designed for paired low-resolution RGB (LR-RGB) and high-resolution panchromatic (HR-PAN) images from satellite sensors with Rational Polynomial Cameras (RPC)
To support the generalization ability of psRPF across scenes, we adopt projection loss to ensure strong geometry self-supervision.
arXiv Detail & Related papers (2024-06-22T02:02:32Z) - DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes.
Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
arXiv Detail & Related papers (2024-06-17T21:15:13Z) - MuRF: Multi-Baseline Radiance Fields [117.55811938988256]
We present Multi-Baseline Radiance Fields (MuRF), a feed-forward approach to solving sparse view synthesis.
MuRF achieves state-of-the-art performance across multiple different baseline settings.
We also show promising zero-shot generalization abilities on the Mip-NeRF 360 dataset.
arXiv Detail & Related papers (2023-12-07T18:59:56Z) - MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane
Images Blending [61.45757368117578]
This paper presents a flexible representation of neural radiance fields based on multi-plane images (MPI)
MPI is widely used in NeRF learning for its simple definition, easy calculation, and powerful ability to represent unbounded scenes.
We show that MPI can synthesize high-quality novel views of complex scenes with diverse camera distributions and view directions.
arXiv Detail & Related papers (2023-09-30T04:36:43Z) - Multi-Plane Neural Radiance Fields for Novel View Synthesis [5.478764356647437]
Novel view synthesis is a long-standing problem that revolves around rendering frames of scenes from novel camera viewpoints.
In this work, we examine the performance, generalization, and efficiency of single-view multi-plane neural radiance fields.
We propose a new multiplane NeRF architecture that accepts multiple views to improve the synthesis results and expand the viewing range.
arXiv Detail & Related papers (2023-03-03T06:32:55Z) - CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural
Representations [77.90883737693325]
This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views.
This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively.
In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
arXiv Detail & Related papers (2022-09-02T17:44:50Z) - Urban Radiance Fields [77.43604458481637]
We perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments.
Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings.
Each of these three extensions provides significant performance improvements in experiments on Street View data.
arXiv Detail & Related papers (2021-11-29T15:58:16Z) - MVSNeRF: Fast Generalizable Radiance Field Reconstruction from
Multi-View Stereo [52.329580781898116]
We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis.
Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference.
arXiv Detail & Related papers (2021-03-29T13:15:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.