DISORF: A Distributed Online 3D Reconstruction Framework for Mobile Robots

• URL: http://arxiv.org/abs/2403.00228v3
• Date: Fri, 2 Aug 2024 05:17:10 GMT
• Title: DISORF: A Distributed Online 3D Reconstruction Framework for Mobile Robots
• Authors: Chunlin Li, Hanrui Fan, Xiaorui Huang, Ruofan Liang, Sankeerth Durvasula, Nandita Vijaykumar,
• Abstract summary: DISORF is a framework to enable online 3D reconstruction and visualization of scenes captured by resource-constrained mobile robots and edge devices. We leverage on-device SLAM systems to generate poseds and transmit them to remote servers that can perform high-quality 3D reconstruction and visualization at runtime. We propose a novel shifted exponential frame sampling method that addresses this challenge for online training.
• Score: 4.683651138674254
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a framework, DISORF, to enable online 3D reconstruction and visualization of scenes captured by resource-constrained mobile robots and edge devices. To address the limited computing capabilities of edge devices and potentially limited network availability, we design a framework that efficiently distributes computation between the edge device and the remote server. We leverage on-device SLAM systems to generate posed keyframes and transmit them to remote servers that can perform high-quality 3D reconstruction and visualization at runtime by leveraging recent advances in neural 3D methods. We identify a key challenge with online training where naive image sampling strategies can lead to significant degradation in rendering quality. We propose a novel shifted exponential frame sampling method that addresses this challenge for online training. We demonstrate the effectiveness of our framework in enabling high-quality real-time reconstruction and visualization of unknown scenes as they are captured and streamed from cameras in mobile robots and edge devices.

Related papers

• Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey [171.72616707259306]
  3D reconstruction and view synthesis are foundational problems in computer vision, graphics, and immersive technologies such as augmented reality (AR), virtual reality (VR), and digital twins.<n>Recent advances in feed-forward approaches, driven by deep learning, have revolutionized this field by enabling fast and generalizable 3D reconstruction and view synthesis.
  arXiv  Detail & Related papers  (2025-07-19T06:13:25Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [63.29645501232935]
  This paper evaluates single-view mesh reconstruction models for creating digital twin environments in robot manipulation.<n>We establish benchmarking criteria for 3D reconstruction in robotics contexts.<n>Despite success on computer vision benchmarks, existing approaches fail to meet robotics-specific requirements.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• A Survey of 3D Reconstruction with Event Cameras [16.103940503726022]
  Event cameras produce sparse yet temporally dense data streams, enabling robust and accurate 3D reconstruction.<n>These capabilities offer substantial promise for transformative applications across various fields, including autonomous driving, robotics, aerial navigation, and immersive virtual reality.<n>This survey aims to serve as an essential reference and provides a clear and motivating roadmap toward advancing the state of the art in event-driven 3D reconstruction.
  arXiv  Detail & Related papers  (2025-05-13T11:04:04Z)
• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [63.21396416244634]
  VideoLifter is a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis. It significantly accelerates the reconstruction process, reducing training time by over 82% while holding better visual quality than current SOTA methods.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)
• DeforHMR: Vision Transformer with Deformable Cross-Attention for 3D Human Mesh Recovery [2.1653492349540784]
  DeforHMR is a novel regression-based monocular HMR framework designed to enhance the prediction of human pose parameters. DeforHMR leverages a novel query-agnostic deformable cross-attention mechanism within the transformer decoder. It achieves state-of-the-art performance for single-frame regression-based methods on the widely used 3D HMR benchmarks 3DPW and RICH.
  arXiv  Detail & Related papers  (2024-11-18T00:46:59Z)
• Federated Multi-View Synthesizing for Metaverse [52.59476179535153]
  The metaverse is expected to provide immersive entertainment, education, and business applications. Virtual reality (VR) transmission over wireless networks is data- and computation-intensive. We have developed a novel multi-view synthesizing framework that can efficiently provide synthesizing, storage, and communication resources for wireless content delivery in the metaverse.
  arXiv  Detail & Related papers  (2023-12-18T13:51:56Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• NSLF-OL: Online Learning of Neural Surface Light Fields alongside Real-time Incremental 3D Reconstruction [0.76146285961466]
  The paper proposes a novel Neural Surface Light Fields model that copes with the small range of view directions while producing a good result in unseen directions. Our model learns online the Neural Surface Light Fields (NSLF) aside from real-time 3D reconstruction with a sequential data stream as the shared input. In addition to online training, our model also provides real-time rendering after completing the data stream for visualization.
  arXiv  Detail & Related papers  (2023-04-29T15:41:15Z)
• Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer [26.21320177775571]
  We propose an end-to-end pose estimation framework that is capable of online camera-to-robot calibration and a self-supervised training method. Our framework combines deep learning and geometric vision for solving the robot pose, and the pipeline is fully differentiable.
  arXiv  Detail & Related papers  (2023-02-28T05:55:42Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device [53.323878851563414]
  We propose a compiler-aware unified framework incorporating network enhancement and pruning search with the reinforcement learning techniques. Specifically, a generator Recurrent Neural Network (RNN) is employed to provide the unified scheme for both network enhancement and pruning search automatically. The proposed framework achieves real-time 3D object detection on mobile devices with competitive detection performance.
  arXiv  Detail & Related papers  (2020-12-26T19:41:15Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)
• A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data [3.4806267677524896]
  We propose a method to deploy state of the art neural networks for real time 3D object localization on augmented reality devices. We focus on fast 2D detection approaches which are extracting the 3D pose of the object fast and accurately by using only 2D input. For the 6D annotation of 2D images, we developed an annotation tool, which is, to our knowledge, the first open source tool to be available.
  arXiv  Detail & Related papers  (2020-01-16T09:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.