Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera

• URL: http://arxiv.org/abs/2511.18174v1
• Date: Sat, 22 Nov 2025 19:57:46 GMT
• Title: Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera
• Abstract summary: Unified Spherical Frontend (USF) is a lens-agnostic framework that transforms images from any camera into a unit-sphere representation via ray-direction correspondences.<n>USF processes high-resolution spherical imagery efficiently and maintains less than 1% performance drop under random test-time rotations.
• Score: 12.448357304482668
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Modern perception increasingly relies on fisheye, panoramic, and other wide field-of-view (FoV) cameras, yet most pipelines still apply planar CNNs designed for pinhole imagery on 2D grids, where image-space neighborhoods misrepresent physical adjacency and models are sensitive to global rotations. Frequency-domain spherical CNNs partially address this mismatch but require costly spherical harmonic transforms that constrain resolution and efficiency. We introduce the Unified Spherical Frontend (USF), a lens-agnostic framework that transforms images from any calibrated camera into a unit-sphere representation via ray-direction correspondences, and performs spherical resampling, convolution, and pooling directly in the spatial domain. USF is modular: projection, location sampling, interpolation, and resolution control are fully decoupled. Its distance-only spherical kernels offer configurable rotation-equivariance (mirroring translation-equivariance in planar CNNs) while avoiding harmonic transforms entirely. We compare standard planar backbones with their spherical counterparts across classification, detection, and segmentation tasks on synthetic (Spherical MNIST) and real-world datasets (PANDORA, Stanford 2D-3D-S), and stress-test robustness to extreme lens distortions, varying FoV, and arbitrary rotations. USF processes high-resolution spherical imagery efficiently and maintains less than 1% performance drop under random test-time rotations, even without rotational augmentation, and even enables zero-shot generalization from one lens type to unseen wide-FoV lenses with minimal performance degradation.

Related papers

• OmniPoint: Universal Monocular Metric Pointcloud from Any Camera [79.24417597712802]
  We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors.<n>To overcome projection rigidity, OmniPoint abandons conventional planar depth regression.<n>It instead adopts a decoupled ray and distance representation alongside a decoupled training objective.
  arXiv  Detail & Related papers  (2026-09-08T19:45:58Z)
• Sphere-VIO: Fast and Robust Visual-Inertial Odometry via Unified Spherical Representation for Heterogeneous Multi-Camera Systems [11.777651307702174]
  Multi-camera visual-inertial odometry (VIO) overcomes the inherent limitations of pure visual systems by expanding the field of view.<n>We present Sphere-VIO, a lightweight filter-based VIO framework with unified spherical representation for heterogeneous multi-camera systems.<n>We show that Sphere-VIO achieves superior trade-offs between accuracy, robustness, efficiency, and cross-camera generality.
  arXiv  Detail & Related papers  (2026-06-29T07:46:08Z)
• UniTriSplat: A Unified 3D Gaussian Splatting Framework with Uniform Spherical Rasterization for Universal Cameras [23.37105294238413]
  UniTriSplat is a unified 3DGS framework for universal cameras that reformulates Gaussian splatting on the unit sphere via HEALPix discretization.<n>We derive the forward rendering and gradient propagation of Gaussians directly in the spherical radian domain, yielding uniform optimization behavior from narrow-FoV images to full 360-degree panoramas.
  arXiv  Detail & Related papers  (2026-06-29T05:14:48Z)
• SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision [15.658046416731098]
  We introduce SpectralSplats, a robust tracking framework that shifts the gradient objective from the spatial to the frequency domain.<n>We demonstrate that SpectralSplats acts as a seamless, drop-in replacement for spatial losses across diverse deformation parameterizations.
  arXiv  Detail & Related papers  (2026-03-25T07:48:22Z)
• Spherical-GOF: Geometry-Aware Panoramic Gaussian Opacity Fields for 3D Scene Reconstruction [17.20174890101058]
  We present Spherical-GOF, an omnidirectional rendering framework built upon Gaussian Opacity Fields (GOF)<n>GOF performs GOF ray sampling directly on the unit sphere in spherical space, enabling consistent ray-Gaussian interactions for panoramic rendering.<n>Experiments on standard panoramic benchmarks demonstrate competitive photometric quality and substantially improved consistency.
  arXiv  Detail & Related papers  (2026-03-09T15:35:56Z)
• FLIGHT: Fibonacci Lattice-based Inference for Geometric Heading in real-Time [7.517221623631364]
  Estimating camera motion from monocular video is a fundamental problem in computer vision.<n>Existing methods that recover the camera's heading under known rotation tend to perform well in low-noise, low-outlier conditions.<n>We propose a novel generalization of the Hough transform on the unit sphere to estimate the camera's heading.
  arXiv  Detail & Related papers  (2026-02-26T15:27:49Z)
• HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis [59.25751939710903]
  We propose a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos.<n>Our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately.<n>Results highlight a practical and scalable solution for EVS in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-24T03:54:40Z)
• AlignDiff: Learning Physically-Grounded Camera Alignment via Diffusion [0.5277756703318045]
  We introduce a novel framework that addresses camera intrinsic and extrinsic parameters using a generic ray camera model.<n>Unlike previous approaches, AlignDiff shifts focus from semantic to geometric features, enabling more accurate modeling of local distortions.<n>Our experiments demonstrate that the proposed method significantly reduces the angular error of estimated ray bundles by 8.2 degrees and overall calibration accuracy, outperforming existing approaches on challenging, real-world datasets.
  arXiv  Detail & Related papers  (2025-03-27T14:59:59Z)
• S-R2D2: a spherical extension of the R2D2 deep neural network series paradigm for wide-field radio-interferometric imaging [0.0]
  Recently, the R2D2 paradigm, standing for ''Residual-to-Residual DNN series for high-Dynamic-range imaging'', was introduced for image formation in Radio Interferometry (RI)<n>We propose the spherical-imaging extension S-R2D2 to meet the spherical-imaging requirement of modern telescopes observing wide fields.
  arXiv  Detail & Related papers  (2025-03-03T12:18:23Z)
• 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction [50.07071392673984]
  Existing methods learn 3D rotations parametrized in the spatial domain using angles or quaternions. We propose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression. Our method achieves state-of-the-art results on benchmarks such as ModelNet10-SO(3) and PASCAL3D+.
  arXiv  Detail & Related papers  (2024-11-01T12:50:38Z)
• SGFormer: Spherical Geometry Transformer for 360 Depth Estimation [52.23806040289676]
  Panoramic distortion poses a significant challenge in 360 depth estimation.<n>We propose a spherical geometry transformer, named SGFormer, to address the above issues.<n>We also present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions.
  arXiv  Detail & Related papers  (2024-04-23T12:36:24Z)
• SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model [63.685132323224124]
  Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges. Experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.
  arXiv  Detail & Related papers  (2024-03-15T06:26:46Z)
• Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields [36.09829614806658]
  We propose L2G-NeRF, a Local-to-Global registration method for Neural Radiance Fields. Pixel-wise local alignment is learned in an unsupervised way via a deep network. Our method outperforms the current state-of-the-art in terms of high-fidelity reconstruction and resolving large camera pose misalignment.
  arXiv  Detail & Related papers  (2022-11-21T14:43:16Z)
• OSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression [59.58879331876508]
  We study the learning of representation models for omnidirectional images and propose to use the properties of HEALPix uniform sampling of the sphere to redefine the mathematical tools used in deep learning models for omnidirectional images. Our proposed on-the-sphere solution leads to a better compression gain that can save 13.7% of the bit rate compared to similar learned models applied to equirectangular images.
  arXiv  Detail & Related papers  (2021-07-19T22:14:30Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs [2.453627017761322]
  We develop scattering networks constructed on the sphere that provide a powerful representational space for spherical data. By integrating scattering networks as an additional type of layer in the generalized spherical CNN framework, we show how they can be leveraged to scale spherical CNNs to the high resolution data typical of many practical applications.
  arXiv  Detail & Related papers  (2021-02-04T19:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.