EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video

• URL: http://arxiv.org/abs/2409.01807v1
• Date: Tue, 3 Sep 2024 11:40:31 GMT
• Title: EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video
• Abstract summary: We present EPRecon, an efficient real-time panoptic 3D reconstruction framework. We propose a lightweight module to directly estimate scene depth priors in a 3D volume. In addition, to infer richer panoptic features from occupied voxels, EPRecon extracts panoptic features from both voxel features and corresponding image features.
• Score: 6.236130301507863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Panoptic 3D reconstruction from a monocular video is a fundamental perceptual task in robotic scene understanding. However, existing efforts suffer from inefficiency in terms of inference speed and accuracy, limiting their practical applicability. We present EPRecon, an efficient real-time panoptic 3D reconstruction framework. Current volumetric-based reconstruction methods usually utilize multi-view depth map fusion to obtain scene depth priors, which is time-consuming and poses challenges to real-time scene reconstruction. To end this, we propose a lightweight module to directly estimate scene depth priors in a 3D volume for reconstruction quality improvement by generating occupancy probabilities of all voxels. In addition, to infer richer panoptic features from occupied voxels, EPRecon extracts panoptic features from both voxel features and corresponding image features, obtaining more detailed and comprehensive instance-level semantic information and achieving more accurate segmentation results. Experimental results on the ScanNetV2 dataset demonstrate the superiority of EPRecon over current state-of-the-art methods in terms of both panoptic 3D reconstruction quality and real-time inference. Code is available at https://github.com/zhen6618/EPRecon.

Related papers

• DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusion [65.65756111062005]
  textitDecoRec is a system designed to elevate single-view 2D images to a decomposed 3D scene mesh.<n>Our results demonstrate that DecoRec facilitates high-quality single-view scene reconstruction in both geometry and novel synthesis.
  arXiv  Detail & Related papers  (2026-05-16T04:23:48Z)
• ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment [43.63998785971779]
  We propose ReplicateAnyScene, a framework capable of fully automated and zero-shot transformation of casually captured videos into compositional 3D scenes.<n>Our pipeline incorporates a five-stage cascade to extract and structurally align generic priors from vision foundation models.<n>Extensive experiments demonstrate the superiority of our method over existing baselines in generating high-quality compositional 3D scenes.
  arXiv  Detail & Related papers  (2026-04-12T19:42:12Z)
• Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction [50.5449251266956]
  This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
  arXiv  Detail & Related papers  (2026-04-09T17:59:50Z)
• NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction [99.52487968452198]
  NOVA3R is an effective approach for non-pixel-aligned 3D reconstruction from a set of unposed images in a feed-forward manner.<n>It produces physically plausible geometry with fewer duplicated structures in overlapping regions.<n>It outperforms state-of-the-art methods in terms of reconstruction accuracy and completeness.
  arXiv  Detail & Related papers  (2026-03-04T15:36:25Z)
• RealX3D: A Physically-Degraded 3D Benchmark for Multi-view Visual Restoration and Reconstruction [51.07270086169647]
  RealX3D groups corruptions into four families, including illumination, scattering, and blurring, and captures each at multiple severity levels.<n>Each scene includes high-resolution capture, RAW images, and dense laser scans, from which we derive world-scale meshes and metric depth.
  arXiv  Detail & Related papers  (2025-12-29T12:57:19Z)
• SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations [44.53106180688135]
  This work takes on the challenge of reconstructing 3D scenes from sparse or single-view inputs.<n>We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations.<n>Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency.
  arXiv  Detail & Related papers  (2025-05-17T13:05:13Z)
• PE3R: Perception-Efficient 3D Reconstruction [54.730257992806116]
  Perception-Efficient 3D Reconstruction (PE3R) is a novel framework designed to enhance both accuracy and efficiency. The framework achieves a minimum 9-fold speedup in 3D semantic field reconstruction, along with substantial gains in perception accuracy and reconstruction precision.
  arXiv  Detail & Related papers  (2025-03-10T16:29:10Z)
• sshELF: Single-Shot Hierarchical Extrapolation of Latent Features for 3D Reconstruction from Sparse-Views [41.73382885439258]
  Reconstructing outdoor scenes from outward-facing views poses significant challenges due to minimal view overlap. We propose a fast, single-shot pipeline for unbounded-view 3D scene reconstruction via hierarchal extrapolation. We find that latentELF faithfully reconstructs occluded regions, supports real-time rendering, and provides rich features for downstream applications.
  arXiv  Detail & Related papers  (2025-02-06T18:58:45Z)
• Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving [116.10577967146762]
  We propose Driv3R, a framework that directly regresses per-frame point maps from multi-view image sequences. We employ a 4D flow predictor to identify moving objects within the scene to direct our network focus more on reconstructing these dynamic regions. Driv3R outperforms previous frameworks in 4D dynamic scene reconstruction, achieving 15x faster inference speed.
  arXiv  Detail & Related papers  (2024-12-09T18:58:03Z)
• ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model [16.14713604672497]
  ReconX is a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. The proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition. Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency.
  arXiv  Detail & Related papers  (2024-08-29T17:59:40Z)
• ReconFusion: 3D Reconstruction with Diffusion Priors [104.73604630145847]
  We present ReconFusion to reconstruct real-world scenes using only a few photos. Our approach leverages a diffusion prior for novel view synthesis, trained on synthetic and multiview datasets. Our method synthesizes realistic geometry and texture in underconstrained regions while preserving the appearance of observed regions.
  arXiv  Detail & Related papers  (2023-12-05T18:59:58Z)
• FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction [13.157400338544177]
  Recent works on 3D reconstruction from posed images have demonstrated that direct inference of scene-level 3D geometry is feasible using deep neural networks. We propose three effective solutions for improving the fidelity of inference-based 3D reconstructions. Our method, FineRecon, produces smooth and highly accurate reconstructions, showing significant improvements across multiple depth and 3D reconstruction metrics.
  arXiv  Detail & Related papers  (2023-04-04T02:50:29Z)
• VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction [64.09702079593372]
  VolRecon is a novel generalizable implicit reconstruction method with Signed Ray Distance Function (SRDF) On DTU dataset, VolRecon outperforms SparseNeuS by about 30% in sparse view reconstruction and achieves comparable accuracy as MVSNet in full view reconstruction.
  arXiv  Detail & Related papers  (2022-12-15T18:59:54Z)
• PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images [105.29493158036105]
  PETRv2 is a unified framework for 3D perception from multi-view images. We extend the 3D position embedding in PETR for temporal modeling. PETRv2 achieves state-of-the-art performance on 3D object detection and BEV segmentation.
  arXiv  Detail & Related papers  (2022-06-02T19:13:03Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video [41.554961144321474]
  We propose to reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module is used to guide the network to fuse features from previous fragments. Experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both accuracy and speed.
  arXiv  Detail & Related papers  (2021-04-01T17:59:46Z)
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo [52.329580781898116]
  We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference.
  arXiv  Detail & Related papers  (2021-03-29T13:15:23Z)
• SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion [86.77318031029404]
  We propose a framework that performs scene reconstruction and semantic scene completion jointly in an incremental and real-time manner. Our framework relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model.
  arXiv  Detail & Related papers  (2020-10-26T15:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.