RS-NeRF: Neural Radiance Fields from Rolling Shutter Images

• URL: http://arxiv.org/abs/2407.10267v1
• Date: Sun, 14 Jul 2024 16:27:11 GMT
• Title: RS-NeRF: Neural Radiance Fields from Rolling Shutter Images
• Authors: Muyao Niu, Tong Chen, Yifan Zhan, Zhuoxiao Li, Xiang Ji, Yinqiang Zheng,
• Abstract summary: We present RS-NeRF, a method designed to synthesize normal images from novel views using input with RS distortions. This involves a physical model that replicates the image formation process under RS conditions. We further address the inherent shortcomings of the basic RS-NeRF model by delving into the RS characteristics and developing algorithms to enhance its functionality.
• Score: 30.719764073204423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields (NeRFs) have become increasingly popular because of their impressive ability for novel view synthesis. However, their effectiveness is hindered by the Rolling Shutter (RS) effects commonly found in most camera systems. To solve this, we present RS-NeRF, a method designed to synthesize normal images from novel views using input with RS distortions. This involves a physical model that replicates the image formation process under RS conditions and jointly optimizes NeRF parameters and camera extrinsic for each image row. We further address the inherent shortcomings of the basic RS-NeRF model by delving into the RS characteristics and developing algorithms to enhance its functionality. First, we impose a smoothness regularization to better estimate trajectories and improve the synthesis quality, in line with the camera movement prior. We also identify and address a fundamental flaw in the vanilla RS model by introducing a multi-sampling algorithm. This new approach improves the model's performance by comprehensively exploiting the RGB data across different rows for each intermediate camera pose. Through rigorous experimentation, we demonstrate that RS-NeRF surpasses previous methods in both synthetic and real-world scenarios, proving its ability to correct RS-related distortions effectively. Codes and data available: https://github.com/MyNiuuu/RS-NeRF

Related papers

• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields [46.186869281594326]
  We propose a novel rolling shutter bundle adjustment method for neural radiance fields (NeRF) We use the unordered rolling shutter (RS) images to obtain the implicit 3D representation.
  arXiv  Detail & Related papers  (2024-03-15T09:08:27Z)
• NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning [63.39461847093663]
  We propose NeRF-VPT, an innovative method for novel view synthesis to address these challenges. Our proposed NeRF-VPT employs a cascading view prompt tuning paradigm, wherein RGB information gained from preceding rendering outcomes serves as instructive visual prompts for subsequent rendering stages. NeRF-VPT only requires sampling RGB data from previous stage renderings as priors at each training stage, without relying on extra guidance or complex techniques.
  arXiv  Detail & Related papers  (2024-03-02T22:08:10Z)
• CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning [23.080474939586654]
  We propose a novel underlinecamera parameter underlinefree neural radiance field (CF-NeRF) CF-NeRF incrementally reconstructs 3D representations and recovers the camera parameters inspired by incremental structure from motion. Results demonstrate that CF-NeRF is robust to camera rotation and achieves state-of-the-art results without providing prior information and constraints.
  arXiv  Detail & Related papers  (2023-12-14T09:09:31Z)
• CorresNeRF: Image Correspondence Priors for Neural Radiance Fields [45.40164120559542]
  CorresNeRF is a novel method that leverages image correspondence priors computed by off-the-shelf methods to supervise NeRF training. We show that this simple yet effective technique of using correspondence priors can be applied as a plug-and-play module across different NeRF variants.
  arXiv  Detail & Related papers  (2023-12-11T18:55:29Z)
• USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields [7.671858441929298]
  We propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF) USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF. Our algorithm can also be used to recover high-fidelity high frame-rate global shutter video from a sequence of RS images.
  arXiv  Detail & Related papers  (2023-10-04T09:51:58Z)
• RBSR: Efficient and Flexible Recurrent Network for Burst Super-Resolution [57.98314517861539]
  Burst super-resolution (BurstSR) aims at reconstructing a high-resolution (HR) image from a sequence of low-resolution (LR) and noisy images. In this paper, we suggest fusing cues frame-by-frame with an efficient and flexible recurrent network.
  arXiv  Detail & Related papers  (2023-06-30T12:14:13Z)
• SiNeRF: Sinusoidal Neural Radiance Fields for Joint Pose Estimation and Scene Reconstruction [147.9379707578091]
  NeRFmm is the Neural Radiance Fields (NeRF) that deal with Joint Optimization tasks. Despite NeRFmm producing precise scene synthesis and pose estimations, it still struggles to outperform the full-annotated baseline on challenging scenes. We propose Sinusoidal Neural Radiance Fields (SiNeRF) that leverage sinusoidal activations for radiance mapping and a novel Mixed Region Sampling (MRS) for selecting ray batch efficiently.
  arXiv  Detail & Related papers  (2022-10-10T10:47:51Z)
• Learning Adaptive Warping for Real-World Rolling Shutter Correction [52.45689075940234]
  This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
  arXiv  Detail & Related papers  (2022-04-29T05:13:50Z)
• NeRF-SR: High-Quality Neural Radiance Fields using Super-Sampling [82.99453001445478]
  We present NeRF-SR, a solution for high-resolution (HR) novel view synthesis with mostly low-resolution (LR) inputs. Our method is built upon Neural Radiance Fields (NeRF) that predicts per-point density and color with a multi-layer perceptron.
  arXiv  Detail & Related papers  (2021-12-03T07:33:47Z)
• NeRF--: Neural Radiance Fields Without Known Camera Parameters [31.01560143595185]
  This paper tackles the problem of novel view synthesis (NVS) from 2D images without known camera poses and intrinsics. We propose an end-to-end framework, termed NeRF--, for training NeRF models given only RGB images.
  arXiv  Detail & Related papers  (2021-02-14T03:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.