MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction

• URL: http://arxiv.org/abs/2308.08741v2
• Date: Thu, 24 Aug 2023 15:43:17 GMT
• Title: MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction
• Authors: Yijie Tang, Jiazhao Zhang, Zhinan Yu, He Wang, Kai Xu
• Abstract summary: We introduce a robust and scalable online RGB-D reconstruction method based on a novel neural implicit representation -- multi-implicit-submap. In our method, neural submaps are incrementally allocated alongside the scanning trajectory and efficiently learned with local neural bundle adjustments. For the first time, randomized optimization is made possible in neural tracking with several key designs to the learning process, enabling efficient and robust tracking even under fast camera motions.
• Score: 15.853932110058585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce MIPS-Fusion, a robust and scalable online RGB-D reconstruction method based on a novel neural implicit representation -- multi-implicit-submap. Different from existing neural RGB-D reconstruction methods lacking either flexibility with a single neural map or scalability due to extra storage of feature grids, we propose a pure neural representation tackling both difficulties with a divide-and-conquer design. In our method, neural submaps are incrementally allocated alongside the scanning trajectory and efficiently learned with local neural bundle adjustments. The submaps can be refined individually in a back-end optimization and optimized jointly to realize submap-level loop closure. Meanwhile, we propose a hybrid tracking approach combining randomized and gradient-based pose optimizations. For the first time, randomized optimization is made possible in neural tracking with several key designs to the learning process, enabling efficient and robust tracking even under fast camera motions. The extensive evaluation demonstrates that our method attains higher reconstruction quality than the state of the arts for large-scale scenes and under fast camera motions.

Related papers

• FMapping: Factorized Efficient Neural Field Mapping for Real-Time Dense RGB SLAM [3.6985351289638957]
  We introduce FMapping, an efficient neural field mapping framework that facilitates the continuous estimation of a colorized point cloud map in real-time dense RGB SLAM. We propose an effective factorization scheme for scene representation and introduce a sliding window strategy to reduce the uncertainty for scene reconstruction.
  arXiv  Detail & Related papers  (2023-06-01T11:51:46Z)
• Towards Theoretically Inspired Neural Initialization Optimization [66.04735385415427]
  We propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. We show that both the training and test performance of a network can be improved by maximizing GradCosine under norm constraint. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost.
  arXiv  Detail & Related papers  (2022-10-12T06:49:16Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)
• Deep Unfolding of the DBFB Algorithm with Application to ROI CT Imaging with Limited Angular Density [15.143939192429018]
  This paper presents a new method for reconstructing regions of interest (ROI) from a limited number of computed (CT) measurements. Deep methods are fast, and they can reach high reconstruction quality by leveraging information from datasets. We introduce an unfolding neural network called UDBFB designed for ROI reconstruction from limited data.
  arXiv  Detail & Related papers  (2022-09-27T09:10:57Z)
• Learning Optimal K-space Acquisition and Reconstruction using Physics-Informed Neural Networks [46.751292014516025]
  Deep neural networks have been applied to reconstruct undersampled k-space data and have shown improved reconstruction performance. This work proposes a novel framework to learn k-space sampling trajectories by considering it as an Ordinary Differential Equation (ODE) problem. Experiments were conducted on different in-viv datasets (textite.g., brain and knee images) acquired with different sequences.
  arXiv  Detail & Related papers  (2022-04-05T20:28:42Z)
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo [97.07453889070574]
  We present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors. We show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes.
  arXiv  Detail & Related papers  (2021-09-02T17:54:31Z)
• Deep Amended Gradient Descent for Efficient Spectral Reconstruction from Single RGB Images [42.26124628784883]
  We propose a compact, efficient, and end-to-end learning-based framework, namely AGD-Net. We first formulate the problem explicitly based on the classic gradient descent algorithm. AGD-Net can improve the reconstruction quality by more than 1.0 dB on average.
  arXiv  Detail & Related papers  (2021-08-12T05:54:09Z)
• A Deep-Unfolded Reference-Based RPCA Network For Video Foreground-Background Separation [86.35434065681925]
  This paper proposes a new deep-unfolding-based network design for the problem of Robust Principal Component Analysis (RPCA) Unlike existing designs, our approach focuses on modeling the temporal correlation between the sparse representations of consecutive video frames. Experimentation using the moving MNIST dataset shows that the proposed network outperforms a recently proposed state-of-the-art RPCA network in the task of video foreground-background separation.
  arXiv  Detail & Related papers  (2020-10-02T11:40:09Z)
• Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting [60.97465664419395]
  We develop a significant and surprising extension of the splitting descent framework that addresses the local optimality issue. By simply allowing both positive and negative weights during splitting, we can eliminate the appearance of splitting stability in S2D. We verify our method on various challenging benchmarks such as CIFAR-100, ImageNet and ModelNet40, on which we outperform S2D and other advanced methods on learning accurate and energy-efficient neural networks.
  arXiv  Detail & Related papers  (2020-03-23T17:09:27Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.