A Portable Multiscopic Camera for Novel View and Time Synthesis in Dynamic Scenes

• URL: http://arxiv.org/abs/2208.14433v1
• Date: Tue, 30 Aug 2022 17:53:17 GMT
• Title: A Portable Multiscopic Camera for Novel View and Time Synthesis in Dynamic Scenes
• Authors: Tianjia Zhang, Yuen-Fui Lau, and Qifeng Chen
• Abstract summary: We present a portable multiscopic camera system with a dedicated model for novel view and time synthesis in dynamic scenes. Our goal is to render high-quality images for a dynamic scene from any viewpoint at any time using our portable multiscopic camera.
• Score: 42.00094186447837
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a portable multiscopic camera system with a dedicated model for novel view and time synthesis in dynamic scenes. Our goal is to render high-quality images for a dynamic scene from any viewpoint at any time using our portable multiscopic camera. To achieve such novel view and time synthesis, we develop a physical multiscopic camera equipped with five cameras to train a neural radiance field (NeRF) in both time and spatial domains for dynamic scenes. Our model maps a 6D coordinate (3D spatial position, 1D temporal coordinate, and 2D viewing direction) to view-dependent and time-varying emitted radiance and volume density. Volume rendering is applied to render a photo-realistic image at a specified camera pose and time. To improve the robustness of our physical camera, we propose a camera parameter optimization module and a temporal frame interpolation module to promote information propagation across time. We conduct experiments on both real-world and synthetic datasets to evaluate our system, and the results show that our approach outperforms alternative solutions qualitatively and quantitatively. Our code and dataset are available at https://yuenfuilau.github.io.

Related papers

• Towards 3D Vision with Low-Cost Single-Photon Cameras [24.711165102559438]
  We present a method for reconstructing 3D shape of arbitrary Lambertian objects based on measurements by miniature, energy-efficient, low-cost single-photon cameras. Our work draws a connection between image-based modeling and active range scanning and is a step towards 3D vision with single-photon cameras.
  arXiv  Detail & Related papers  (2024-03-26T15:40:05Z)
• Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images [65.41966114373373]
  We present an improved solution to the neural image-based rendering problem in computer vision. The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
  arXiv  Detail & Related papers  (2023-11-08T08:18:23Z)
• Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting [8.078460597825142]
  Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. We propose to approximate the underlying-temporal rendering volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics.
  arXiv  Detail & Related papers  (2023-10-16T17:57:43Z)
• DynaMoN: Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields [71.94156412354054]
  We propose DynaMoN to handle dynamic content for initial camera pose estimation and statics-focused ray sampling for fast and accurate novel-view synthesis. Our novel iterative learning scheme switches between training the NeRF and updating the pose parameters for an improved reconstruction and trajectory estimation quality.
  arXiv  Detail & Related papers  (2023-09-16T08:46:59Z)
• An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras [10.591040194296315]
  We present an asynchronous linear filter architecture, fusing event and frame camera data, for HDR video reconstruction and spatial convolution. The proposed AKF pipeline outperforms other state-of-the-art methods in both absolute intensity error (69.4% reduction) and image similarity indexes (average 35.5% improvement)
  arXiv  Detail & Related papers  (2023-09-03T12:37:59Z)
• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields [99.57774680640581]
  We present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. We propose to decompose the 4D space according to temporal characteristics. Points in the 4D space are associated with probabilities belonging to three categories: static, deforming, and new areas.
  arXiv  Detail & Related papers  (2022-10-28T07:11:05Z)
• Acquiring a Dynamic Light Field through a Single-Shot Coded Image [12.615509935080434]
  We propose a method for compressively acquiring a dynamic light field (a 5-D volume) through a single-shot coded image (a 2-D measurement) We designed an imaging model that synchronously applies aperture coding and pixel-wise exposure coding within a single exposure time. The observed image is then fed to a convolutional neural network (CNN) for light-field reconstruction, which is jointly trained with the camera-side coding patterns.
  arXiv  Detail & Related papers  (2022-04-26T06:00:02Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras [63.186486240525554]
  DeepMultiCap is a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models.
  arXiv  Detail & Related papers  (2021-05-01T14:32:13Z)
• Event-based Stereo Visual Odometry [42.77238738150496]
  We present a solution to the problem of visual odometry from the data acquired by a stereo event-based camera rig. We seek to maximize thetemporal consistency of stereo event-based data while using a simple and efficient representation.
  arXiv  Detail & Related papers  (2020-07-30T15:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.