ACT-R: Adaptive Camera Trajectories for 3D Reconstruction from Single Image

• URL: http://arxiv.org/abs/2505.08239v1
• Date: Tue, 13 May 2025 05:31:59 GMT
• Title: ACT-R: Adaptive Camera Trajectories for 3D Reconstruction from Single Image
• Authors: Yizhi Wang, Mingrui Zhao, Ali Mahdavi-Amiri, Hao Zhang,
• Abstract summary: We introduce adaptive view planning to multi-view synthesis.<n>We generate a sequence of views, leveraging temporal consistency to enhance 3D coherence.<n>Our method significantly improves 3D reconstruction over SOTA on the unseen GSO dataset.
• Score: 12.942796503696194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce adaptive view planning to multi-view synthesis, aiming to improve both occlusion revelation and 3D consistency for single-view 3D reconstruction. Instead of generating an unordered set of views independently or simultaneously, we generate a sequence of views, leveraging temporal consistency to enhance 3D coherence. Most importantly, our view sequence is not determined by a pre-determined camera setup. Instead, we compute an adaptive camera trajectory (ACT), specifically, an orbit of camera views, which maximizes the visibility of occluded regions of the 3D object to be reconstructed. Once the best orbit is found, we feed it to a video diffusion model to generate novel views around the orbit, which in turn, are passed to a multi-view 3D reconstruction model to obtain the final reconstruction. Our multi-view synthesis pipeline is quite efficient since it involves no run-time training/optimization, only forward inferences by applying the pre-trained models for occlusion analysis and multi-view synthesis. Our method predicts camera trajectories that reveal occlusions effectively and produce consistent novel views, significantly improving 3D reconstruction over SOTA on the unseen GSO dataset, both quantitatively and qualitatively.

Related papers

• SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations [44.53106180688135]
  This work takes on the challenge of reconstructing 3D scenes from sparse or single-view inputs.<n>We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations.<n>Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency.
  arXiv  Detail & Related papers  (2025-05-17T13:05:13Z)
• HORT: Monocular Hand-held Objects Reconstruction with Transformers [61.36376511119355]
  Reconstructing hand-held objects in 3D from monocular images is a significant challenge in computer vision.<n>We propose a transformer-based model to efficiently reconstruct dense 3D point clouds of hand-held objects.<n>Our method achieves state-of-the-art accuracy with much faster inference speed, while generalizing well to in-the-wild images.
  arXiv  Detail & Related papers  (2025-03-27T09:45:09Z)
• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization.<n>We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• MUSt3R: Multi-view Network for Stereo 3D Reconstruction [11.61182864709518]
  We propose an extension of DUSt3R from pairs to multiple views, that addresses all aforementioned concerns.<n>We entail the model with a multi-layer memory mechanism which allows to reduce the computational complexity.<n>The framework is designed to perform 3D reconstruction both offline and online, and hence can be seamlessly applied to SfM and visual SLAM scenarios.
  arXiv  Detail & Related papers  (2025-03-03T15:36:07Z)
• Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model [15.892685514932323]
  We introduce Plane-DUSt3R, a novel method for multi-view room layout estimation.<n>Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes.<n>By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results.
  arXiv  Detail & Related papers  (2025-02-24T02:14:19Z)
• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [63.21396416244634]
  VideoLifter is a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis.<n>It significantly accelerates the reconstruction process, reducing training time by over 82% while holding better visual quality than current SOTA methods.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)
• Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis [25.898616784744377]
  Given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D.<n>We propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them.
  arXiv  Detail & Related papers  (2024-12-04T18:59:24Z)
• PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence [3.61512056914095]
  We present PreF3R, Pose-Free Feed-forward 3D Reconstruction from an image sequence of variable length. PreF3R removes the need for camera calibration and reconstructs the 3D Gaussian field within a canonical coordinate frame directly from a sequence of unposed images.
  arXiv  Detail & Related papers  (2024-11-25T19:16:29Z)
• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis [63.169364481672915]
  We propose textbfViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames.
  arXiv  Detail & Related papers  (2024-09-03T16:53:19Z)
• MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation [54.27399121779011]
  We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images. We show that our approach can yield more accurate synthesis compared to recent state-of-the-art, including distillation-based 3D inference and prior multi-view generation methods.
  arXiv  Detail & Related papers  (2024-04-04T17:59:57Z)
• Learning to Render Novel Views from Wide-Baseline Stereo Pairs [26.528667940013598]
  We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. Existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry. We propose an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray.
  arXiv  Detail & Related papers  (2023-04-17T17:40:52Z)
• Enhanced Stable View Synthesis [86.69338893753886]
  We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging.
  arXiv  Detail & Related papers  (2023-03-30T01:53:14Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Novel View Synthesis with Diffusion Models [56.55571338854636]
  We present 3DiM, a diffusion model for 3D novel view synthesis. It is able to translate a single input view into consistent and sharp completions across many views. 3DiM can generate multiple views that are 3D consistent using a novel technique called conditioning.
  arXiv  Detail & Related papers  (2022-10-06T16:59:56Z)
• Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild [49.672487902268706]
  We present a framework that jointly estimates camera temporal alignment and 3D point triangulation. We reconstruct 3D motion trajectories of human bodies in events captured by multiple unsynchronized and unsynchronized video cameras.
  arXiv  Detail & Related papers  (2020-07-24T23:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.