FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception

• URL: http://arxiv.org/abs/2604.10391v1
• Date: Sun, 12 Apr 2026 00:46:51 GMT
• Title: FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception
• Abstract summary: Vision foundation models (VFMs) and Bird's Eye View (BEV) representation assume the rectilinear geometry of pinhole cameras.<n>ours is a lightweight framework that adapts frozen VFMs to fisheye geometry through two components.
• Score: 6.114308991453516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision foundation models (VFMs) and Bird's Eye View (BEV) representation have advanced visual perception substantially, yet their internal spatial representations assume the rectilinear geometry of pinhole cameras. Fisheye cameras, widely deployed on production autonomous vehicles for their surround-view coverage, exhibit severe radial distortion that renders these representations geometrically inconsistent. At the same time, the scarcity of large-scale fisheye annotations makes retraining foundation models from scratch impractical. We present \ours, a lightweight framework that adapts frozen VFMs to fisheye geometry through two components: a frozen DINOv2 backbone with Low-Rank Adaptation (LoRA) that transfers rich self-supervised features to fisheye without task-specific pretraining, and Fisheye Rotary Position Embedding (FishRoPE), which reparameterizes the attention mechanism in the spherical coordinates of the fisheye projection so that both self-attention and cross-attention operate on angular separation rather than pixel distance. FishRoPE is architecture-agnostic, introduces negligible computational overhead, and naturally reduces to the standard formulation under pinhole geometry. We evaluate \ours on WoodScape 2D detection (54.3 mAP) and SynWoodScapes BEV segmentation (65.1 mIoU), where it achieves state-of-the-art results on both benchmarks.

Related papers

• OmniPoint: Universal Monocular Metric Pointcloud from Any Camera [79.24417597712802]
  We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors.<n>To overcome projection rigidity, OmniPoint abandons conventional planar depth regression.<n>It instead adopts a decoupled ray and distance representation alongside a decoupled training objective.
  arXiv  Detail & Related papers  (2026-09-08T19:45:58Z)
• Geometry-Aware Fisheye-LiDAR Fusion for Robust 3D Object Detection in Low-Overlap Setups [8.422269543220251]
  We propose a Geometry-Aware Hybrid Fusion (GA-HF) framework that accounts for fisheye geometry and BEV feature distortion.<n>GA-HF is evaluated on three benchmarks: KITTI-360, Dur360BEV, and Fisheye3DOD datasets.
  arXiv  Detail & Related papers  (2026-06-07T21:26:00Z)
• Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses [5.987299558427844]
  Fisheye3R is a novel adaptation framework that extends multi-view 3D reconstruction foundation models to accommodate fisheye inputs without performance regression on perspective images.<n>Our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images.
  arXiv  Detail & Related papers  (2026-03-30T18:18:56Z)
• Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras [5.227238848375275]
  We introduce a multi-view BEV detection benchmark with mixed cameras by converting KITTI-360 into nuScenes format.<n>Our study encompasses three adaptations: rectification for zero-shot evaluation and fine-tuning of nuScenes-trained models.<n>We demonstrate that projection-free architectures are inherently more robust and effective against fisheye distortion than other VTMs.
  arXiv  Detail & Related papers  (2026-03-29T19:10:34Z)
• Exploring Surround-View Fisheye Camera 3D Object Detection [44.09164098759971]
  We investigate the technical feasibility of implementing end-to-end 3D object detection with surround-view fisheye camera system.<n>We develop two methods that incorporate the unique geometry of fisheye images into mainstream detection frameworks.<n> Experiments on Fisheye3DOD show that our fisheye-compatible modeling improves accuracy by up to 6.2% over baseline methods.
  arXiv  Detail & Related papers  (2025-11-24T02:28:56Z)
• PFDepth: Heterogeneous Pinhole-Fisheye Joint Depth Estimation via Distortion-aware Gaussian-Splatted Volumetric Fusion [61.6340987158734]
  We present the first pinhole-fisheye framework for heterogeneous multi-view depth estimation, PFDepth.<n> PFDepth employs a unified architecture capable of processing arbitrary combinations of pinhole and fisheye cameras with varied intrinsics and extrinsics.<n>We show that PFDepth sets a state-of-the-art performance on KITTI-360 and RealHet datasets over current mainstream depth networks.
  arXiv  Detail & Related papers  (2025-09-30T09:38:59Z)
• FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras [12.001699443894504]
  We propose FishBEV, a novel BEV segmentation framework specifically tailored for fisheye cameras.<n>This framework introduces three complementary innovations, including a Distortion-Resilient Multi-scale Extraction (DRME) backbone that learns robust features under distortion while preserving scale consistency.<n>Experiments on the Synwoodscapes dataset demonstrate that FishBEV consistently outperforms SOTA baselines, regarding the performance evaluation of FishBEV on the surround-view fisheye BEV segmentation tasks.
  arXiv  Detail & Related papers  (2025-09-17T04:26:36Z)
• FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views [100.45129752375658]
  We present FLARE, a feed-forward model designed to infer high-quality camera poses and 3D geometry from uncalibrated sparse-view images.<n>Our solution features a cascaded learning paradigm with camera pose serving as the critical bridge, recognizing its essential role in mapping 3D structures onto 2D image planes.
  arXiv  Detail & Related papers  (2025-02-17T18:54:05Z)
• RoFIR: Robust Fisheye Image Rectification Framework Impervious to Optical Center Deviation [88.54817424560056]
  We propose a distortion vector map (DVM) that measures the degree and direction of local distortion. By learning the DVM, the model can independently identify local distortions at each pixel without relying on global distortion patterns. In the pre-training stage, it predicts the distortion vector map and perceives the local distortion features of each pixel. In the fine-tuning stage, it predicts a pixel-wise flow map for deviated fisheye image rectification.
  arXiv  Detail & Related papers  (2024-06-27T06:38:56Z)
• FisheyeEX: Polar Outpainting for Extending the FoV of Fisheye Lens [84.12722334460022]
  Fisheye lens gains increasing applications in computational photography and assisted driving because of its wide field of view (FoV) In this paper, we present a FisheyeEX method that extends the FoV of the fisheye lens by outpainting the invalid regions. The results demonstrate that our approach significantly outperforms the state-of-the-art methods, gaining around 27% more content beyond the original fisheye image.
  arXiv  Detail & Related papers  (2022-06-12T21:38:50Z)
• Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion [51.19260542887099]
  We show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems.
  arXiv  Detail & Related papers  (2020-08-15T02:29:13Z)
• SilhoNet-Fisheye: Adaptation of A ROI Based Object Pose Estimation Network to Monocular Fisheye Images [15.573003283204958]
  We present a novel framework for adapting a ROI-based 6D object pose estimation method to work on full fisheye images. We also contribute a fisheye image dataset, called UWHandles, with 6D object pose and 2D bounding box annotations.
  arXiv  Detail & Related papers  (2020-02-27T19:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.