RayTun3R: Online Camera Adaptation in 3D Foundation Models

• URL: http://arxiv.org/abs/2607.02711v1
• Date: Thu, 02 Jul 2026 18:55:35 GMT
• Title: RayTun3R: Online Camera Adaptation in 3D Foundation Models
• Abstract summary: Recent 3D foundation models provide strong feed-forward depth and pose estimates on pinhole imagery, but degrade sharply under fisheye camera geometry.<n>We show that this failure is partly caused by a pinhole camera bias in the positional encodings of pretrained 3D foundation models.<n>We propose RayTun3R, a lightweight camera adaptation approach.
• Score: 53.97305597591864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent 3D foundation models, such as DUSt3R, MASt3R, VGGT, $π^3$, and Depth Anything 3, provide strong feed-forward depth and pose estimates on pinhole imagery, but degrade sharply under fisheye camera geometry. We show that this failure is partly caused by a pinhole camera bias in the positional encodings of pretrained 3D foundation models, and propose RayTun3R, a lightweight camera adaptation approach. It keeps the pretrained network fixed and adapts only lightweight components tied to token position and camera geometry. RayTun3R learns parameter-efficient residual corrections to absolute and rotary positional encodings, together with parameter-free tokenization and corrections to prediction-grid coordinates that remove residual pinhole assumptions. The resulting adapter contains only 10,752 trainable parameters and can be learned from a short temporal segment using geometric losses. Once adapted, RayTun3R transfers effectively to the remaining frames of the sequence without incurring additional runtime costs. Across diverse fisheye datasets with fields of view from $110^\circ$ to $200^\circ$, our adapter reduces rotation error by $2$-$12\times$ relative to the unadapted model, outperforms LoRA while using $\sim\!14\times$ fewer trainable parameters, improves pose over adaptation-free baselines while avoiding their multi-view inference cost, and remains competitive on depth accuracy.

Related papers

• OmniPoint: Universal Monocular Metric Pointcloud from Any Camera [79.24417597712802]
  We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors.<n>To overcome projection rigidity, OmniPoint abandons conventional planar depth regression.<n>It instead adopts a decoupled ray and distance representation alongside a decoupled training objective.
  arXiv  Detail & Related papers  (2026-09-08T19:45:58Z)
• Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs [72.8641426724502]
  3D localization in Multimodal Large Language Models (MLLMs) is limited by camera intrinsic ambiguity.<n>We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables.<n>Our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale.
  arXiv  Detail & Related papers  (2026-05-19T08:30:21Z)
• Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses [5.987299558427844]
  Fisheye3R is a novel adaptation framework that extends multi-view 3D reconstruction foundation models to accommodate fisheye inputs without performance regression on perspective images.<n>Our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images.
  arXiv  Detail & Related papers  (2026-03-30T18:18:56Z)
• LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models [36.17554088247551]
  LoRA3D is an efficient self-calibration pipeline for 3D geometric foundation models.<n>We leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame.<n>Our method completes the self-calibration process on a $textbfsingle standard GPU within just 5 minutes$.
  arXiv  Detail & Related papers  (2024-12-10T18:45:04Z)
• InstantSplat: Sparse-view Gaussian Splatting in Seconds [91.77050739918037]
  We introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed.<n>InstantSplat employs a self-supervised framework that optimize 3D scene representation and camera poses.<n>It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.
  arXiv  Detail & Related papers  (2024-03-29T17:29:58Z)
• Camera Calibration through Geometric Constraints from Rotation and Projection Matrices [4.100632594106989]
  We propose a novel constraints-based loss for measuring the intrinsic and extrinsic parameters of a camera. Our methodology is a hybrid approach that employs the learning power of a neural network to estimate the desired parameters. Our proposed approach demonstrates improvements across all parameters when compared to the state-of-the-art (SOTA) benchmarks.
  arXiv  Detail & Related papers  (2024-02-13T13:07:34Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• DIME-Net: Neural Network-Based Dynamic Intrinsic Parameter Rectification for Cameras with Optical Image Stabilization System [16.390775530663618]
  We propose a novel neural network-based approach that estimates pose estimation or 3D reconstruction in real-time. We name the proposed Dynamic Intrinsic pose estimation network as DIME-Net and have it implemented and tested on three different mobile devices. In all cases, DIME-Net can reduce reprojection error by at least $64$% indicating that our design is successful.
  arXiv  Detail & Related papers  (2023-03-20T17:45:12Z)
• Uncertainty-Aware Camera Pose Estimation from Points and Lines [101.03675842534415]
  Perspective-n-Point-and-Line (Pn$PL) aims at fast, accurate and robust camera localizations with respect to a 3D model from 2D-3D feature coordinates.
  arXiv  Detail & Related papers  (2021-07-08T15:19:36Z)
• Fisheye Distortion Rectification from Deep Straight Lines [34.61402494687801]
  We present a novel line-aware rectification network (LaRecNet) to address the problem of fisheye distortion rectification. Our model achieves state-of-the-art performance in terms of both geometric accuracy and image quality. In particular, the images rectified by LaRecNet achieve the highest peak signal-to-noise ratio (PSNR) and structure similarity index (SSIM) compared with the groundtruth.
  arXiv  Detail & Related papers  (2020-03-25T13:20:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.