Learning 3D Reconstruction with Priors in Test Time

• URL: http://arxiv.org/abs/2604.03878v1
• Date: Sat, 04 Apr 2026 22:10:28 GMT
• Title: Learning 3D Reconstruction with Priors in Test Time
• Abstract summary: We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors to improve 3D tasks without retraining or modifying pre-trained image-only networks.<n>Our method consistently improves performance over base MVTs by a large margin.
• Score: 38.04707926304637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a test-time framework for multiview Transformers (MVTs) that incorporates priors (e.g., camera poses, intrinsics, and depth) to improve 3D tasks without retraining or modifying pre-trained image-only networks. Rather than feeding priors into the architecture, we cast them as constraints on the predictions and optimize the network at inference time. The optimization loss consists of a self-supervised objective and prior penalty terms. The self-supervised objective captures the compatibility among multi-view predictions and is implemented using photometric or geometric loss between renderings from other views and each view itself. Any available priors are converted into penalty terms on the corresponding output modalities. Across a series of 3D vision benchmarks, including point map estimation and camera pose estimation, our method consistently improves performance over base MVTs by a large margin. On the ETH3D, 7-Scenes, and NRGBD datasets, our method reduces the point-map distance error by more than half compared with the base image-only models. Our method also outperforms retrained prior-aware feed-forward methods, demonstrating the effectiveness of our test-time constrained optimization (TCO) framework for incorporating priors into 3D vision tasks.

Related papers

• JEPADepth: Masked Predictive Representation Learning for Self-Supervised Monocular Depth Estimation [0.0]
  JEPADepth is a self-supervised monocular depth framework that incorporates a complementary training objective.<n>Our method augments a standard photometric pipeline with a masked prediction loss computed in the representation space of a pretrained DINOv3 Vision Transformer encoder.<n>A predictor infers target-region embeddings from visible context-region embeddings under structured masking, and is discarded along with the target encoder at inference time, adding no deployment cost.
  arXiv  Detail & Related papers  (2026-07-29T08:23:46Z)
• PRISM: Feed-Forward Single-Image 3D Reconstruction via Geometric Warp-Residual Modeling [11.124244013253806]
  Reconstructing 3D scenes from a single image is a fundamental challenge in computer vision.<n>Recent methods achieve outstanding performance by leveraging camera-controlled video diffusion models.<n>We propose PRISM, a feed-forward framework that decomposes multi-view latent prediction into a parameter-free geometric prior.
  arXiv  Detail & Related papers  (2026-06-24T05:48:20Z)
• Registration-Free Learnable Multi-View Capture of Faces in Dense Semantic Correspondence [30.38509036690193]
  MOCHI is a multi-view 3D face prediction framework trained without requiring registered training data.<n>Standard point-to-surface distances induce training instabilities and visual artifacts in registration-free settings.<n>We propose pointmap- and normal-based losses instead, which provide smoother gradients and superior reconstruction fidelity.
  arXiv  Detail & Related papers  (2026-05-02T13:59:29Z)
• From Editor to Dense Geometry Estimator [77.21804448599009]
  We introduce textbfFE2E, a framework that adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction.<n>FE2E achieves over 35% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100$times$ data.
  arXiv  Detail & Related papers  (2025-09-04T15:58:50Z)
• How to Use Diffusion Priors under Sparse Views? [29.738350228085928]
  Inline Prior Guided Score Matching is proposed to provide visual supervision over sparse views in 3D reconstruction.<n>We show that our method achieves state-of-the-art reconstruction quality.
  arXiv  Detail & Related papers  (2024-12-03T07:31:54Z)
• VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving [44.91443640710085]
  VisionPAD is a novel self-supervised pre-training paradigm for vision-centric algorithms in autonomous driving.<n>It reconstructs multi-view representations using only images as supervision.<n>It significantly improves performance in 3D object detection, occupancy prediction and map segmentation.
  arXiv  Detail & Related papers  (2024-11-22T03:59:41Z)
• A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision [0.0]
  Self-supervised pre-training for 3D vision has drawn increasing research interest in recent years. We present a unified framework under which various pre-training methods can be investigated. We propose a simple but effective method that jointly pre-trains a 3D encoder and a depth map encoder using contrastive learning.
  arXiv  Detail & Related papers  (2022-07-11T16:44:15Z)
• Multi-initialization Optimization Network for Accurate 3D Human Pose and Shape Estimation [75.44912541912252]
  We propose a three-stage framework named Multi-Initialization Optimization Network (MION) In the first stage, we strategically select different coarse 3D reconstruction candidates which are compatible with the 2D keypoints of input sample. In the second stage, we design a mesh refinement transformer (MRT) to respectively refine each coarse reconstruction result via a self-attention mechanism. Finally, a Consistency Estimation Network (CEN) is proposed to find the best result from mutiple candidates by evaluating if the visual evidence in RGB image matches a given 3D reconstruction.
  arXiv  Detail & Related papers  (2021-12-24T02:43:58Z)
• Riggable 3D Face Reconstruction via In-Network Optimization [58.016067611038046]
  This paper presents a method for riggable 3D face reconstruction from monocular images. It jointly estimates a personalized face rig and per-image parameters including expressions, poses, and illuminations. Experiments demonstrate that our method achieves SOTA reconstruction accuracy, reasonable robustness and generalization ability.
  arXiv  Detail & Related papers  (2021-04-08T03:53:20Z)
• GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator [51.89441403642665]
  6D pose estimation of rigid objects is a long-standing and challenging task in computer vision.<n>Recently, the emergence of deep learning reveals the potential of Convolutional Neural Networks (CNNs) to predict reliable 6D poses.<n>This paper introduces a fully learning-based object pose estimator.
  arXiv  Detail & Related papers  (2021-02-24T09:11:31Z)
• Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training [3.8073142980733]
  We propose a novel framework for monocular 3D objects detection using only RGB images, called KM3D-Net. We design a fully convolutional model to predict object keypoints, dimension, and orientation, and then combine these estimations with perspective geometry constraints to compute position attribute.
  arXiv  Detail & Related papers  (2020-09-02T00:51:51Z)
• Reinforced Axial Refinement Network for Monocular 3D Object Detection [160.34246529816085]
  Monocular 3D object detection aims to extract the 3D position and properties of objects from a 2D input image. Conventional approaches sample 3D bounding boxes from the space and infer the relationship between the target object and each of them, however, the probability of effective samples is relatively small in the 3D space. We propose to start with an initial prediction and refine it gradually towards the ground truth, with only one 3d parameter changed in each step. This requires designing a policy which gets a reward after several steps, and thus we adopt reinforcement learning to optimize it.
  arXiv  Detail & Related papers  (2020-08-31T17:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.