Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

• URL: http://arxiv.org/abs/2603.28896v1
• Date: Mon, 30 Mar 2026 18:18:56 GMT
• Title: Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses
• Abstract summary: Fisheye3R is a novel adaptation framework that extends multi-view 3D reconstruction foundation models to accommodate fisheye inputs without performance regression on perspective images.<n>Our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images.
• Score: 5.987299558427844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Feed-forward foundation models for multi-view 3-dimensional (3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. Their error arises from changes in spatial positions of pixels due to a non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limit generalization. To enable inference on imagery exhibiting high radial distortion, we propose Fisheye3R, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT, $π^3$, and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images.

Related papers

• From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation [6.335915125785494]
  We propose a method to generalize vision foundation models to fisheye cameras.<n>The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX)<n>DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates.
  arXiv  Detail & Related papers  (2026-08-28T03:10:18Z)
• DirectFisheye-GS: Enabling Native Fisheye Input in Gaussian Splatting with Cross-View Joint Optimization [17.91043282211744]
  3D Gaussian Splatting (3DGS) has enabled efficient 3D scene reconstruction from everyday images with real-time, high-fidelity rendering.<n>Fisheye cameras, with their wider field of view (FOV), promise high-quality reconstructions from fewer inputs.
  arXiv  Detail & Related papers  (2026-04-01T09:00:04Z)
• Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras [5.227238848375275]
  We introduce a multi-view BEV detection benchmark with mixed cameras by converting KITTI-360 into nuScenes format.<n>Our study encompasses three adaptations: rectification for zero-shot evaluation and fine-tuning of nuScenes-trained models.<n>We demonstrate that projection-free architectures are inherently more robust and effective against fisheye distortion than other VTMs.
  arXiv  Detail & Related papers  (2026-03-29T19:10:34Z)
• Exploring Surround-View Fisheye Camera 3D Object Detection [44.09164098759971]
  We investigate the technical feasibility of implementing end-to-end 3D object detection with surround-view fisheye camera system.<n>We develop two methods that incorporate the unique geometry of fisheye images into mainstream detection frameworks.<n> Experiments on Fisheye3DOD show that our fisheye-compatible modeling improves accuracy by up to 6.2% over baseline methods.
  arXiv  Detail & Related papers  (2025-11-24T02:28:56Z)
• UniK3D: Universal Camera Monocular 3D Estimation [62.06785782635153]
  We present UniK3D, the first generalizable method for monocular 3D estimation able to model any camera.<n>Our method introduces a spherical 3D representation which allows for better disentanglement of camera and scene geometry.<n>A comprehensive zero-shot evaluation on 13 diverse datasets demonstrates the state-of-the-art performance of UniK3D across 3D, depth, and camera metrics.
  arXiv  Detail & Related papers  (2025-03-20T17:49:23Z)
• FisheyeDepth: A Real Scale Self-Supervised Depth Estimation Model for Fisheye Camera [8.502741852406904]
  We present FisheyeDepth, a self-supervised depth estimation model tailored for fisheye cameras.<n>We incorporate a fisheye camera model into the projection and reprojection stages during training to handle image distortions.<n>We also incorporate real-scale pose information into the geometric projection between consecutive frames, replacing the poses estimated by the conventional pose network.
  arXiv  Detail & Related papers  (2024-09-23T14:31:42Z)
• Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras [17.75361921797657]
  We introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods.
  arXiv  Detail & Related papers  (2024-09-07T07:53:40Z)
• RoFIR: Robust Fisheye Image Rectification Framework Impervious to Optical Center Deviation [88.54817424560056]
  We propose a distortion vector map (DVM) that measures the degree and direction of local distortion. By learning the DVM, the model can independently identify local distortions at each pixel without relying on global distortion patterns. In the pre-training stage, it predicts the distortion vector map and perceives the local distortion features of each pixel. In the fine-tuning stage, it predicts a pixel-wise flow map for deviated fisheye image rectification.
  arXiv  Detail & Related papers  (2024-06-27T06:38:56Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image [91.71077190961688]
  We propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image. We then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot evaluation.
  arXiv  Detail & Related papers  (2022-08-28T16:20:14Z)
• FisheyeEX: Polar Outpainting for Extending the FoV of Fisheye Lens [84.12722334460022]
  Fisheye lens gains increasing applications in computational photography and assisted driving because of its wide field of view (FoV) In this paper, we present a FisheyeEX method that extends the FoV of the fisheye lens by outpainting the invalid regions. The results demonstrate that our approach significantly outperforms the state-of-the-art methods, gaining around 27% more content beyond the original fisheye image.
  arXiv  Detail & Related papers  (2022-06-12T21:38:50Z)
• MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision [72.5863451123577]
  We show how to train a neural model that can perform accurate 3D pose and camera estimation. Our method outperforms both classical bundle adjustment and weakly-supervised monocular 3D baselines.
  arXiv  Detail & Related papers  (2021-08-10T18:39:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.