Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild

• URL: http://arxiv.org/abs/2609.09012v2
• Date: Tue, 15 Sep 2026 02:05:23 GMT
• Title: Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild
• Abstract summary: We introduce Spheriverse, comprising 64,400 temporally aligned spherical image-LiDAR pairs organized into 644 sequences.<n>The dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes.<n>For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence retrieval.
• Score: 29.435475031890523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spherical observations provide global visual context for 3D scene understanding. However, visual information is encoded in an angular domain, whereas the physical world is represented in Cartesian coordinates. This cross-space representation gap complicates geometric correspondence and semantic evidence aggregation. To delve into this challenge, we introduce Spheriverse, comprising 64,400 temporally aligned spherical image-LiDAR pairs organized into 644 sequences. The dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes. We further establish benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, evaluating 30+ methods through overall and scene-wise comparisons. For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence retrieval. Cartesian-Spherical Representation Remodeling (CSRR) incorporates spherical range-azimuth geometry into Cartesian voxel features through region-wise modulation. Spherical Evidence Re-querying (SER) then conditions queries on voxel content and range-height-azimuth geometry to adaptively retrieve relevant semantic evidence from source spherical image features. SphereOcc achieves 13.91% mIoU and 24.65% GeoIoU, yielding relative improvements of 13.9% and 9.3% over the respective best-performing methods, TPVFormer and SurroundOcc. It also ranks first in both metrics across all five scene categories, with consistent advantages across the evaluated spatial partitions and reduced fields of view. The established benchmark and source code will be available at https://feit-feiteng.github.io/Spheriverse.

Related papers

• SphereSOD: Geometry-Structure Coupled Learning for 360 Salient Object Detection [75.77856258186854]
  360 salient object detection aims to accurately segment salient regions across a full field of view.<n>Existing methods mainly focus on compensating projection distortion while overlooking the interaction between panoramic geometry and salient object structure.<n>We propose SphereSOD, an ERP-native framework that couples panoramic geometry with evolving salient structures.
  arXiv  Detail & Related papers  (2026-09-07T14:47:54Z)
• HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulation [54.325346533275074]
  HeRO is a diffusion-based policy that couples geometry and semantics via hierarchical semantic fields.<n>In various tests, HeRO establishes a new state-of-the-art, improving success on Place Dual Shoes by 12.3% and averaging 6.5% gains across six challenging pose-aware tasks.
  arXiv  Detail & Related papers  (2026-02-21T12:29:10Z)
• EDM: Equirectangular Projection-Oriented Dense Kernelized Feature Matching [49.15373858190824]
  We introduce the first learning-based dense matching algorithm, termed Equirectangular Projection-Oriented Kernelized Feature Matching (EDM)<n>Our proposed method achieves notable performance enhancements, with improvements of +26.72 and +42.62 in AUC@5deg on the Matterport3D and Stanford2D3D datasets.
  arXiv  Detail & Related papers  (2025-02-28T03:37:01Z)
• GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction [70.65250036489128]
  3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene. We propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians. GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption.
  arXiv  Detail & Related papers  (2024-05-27T17:59:51Z)
• 3D Scene Geometry Estimation from 360$^\circ$ Imagery: A Survey [1.3654846342364308]
  This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data.
  arXiv  Detail & Related papers  (2024-01-17T14:57:27Z)
• Camera-based 3D Semantic Scene Completion with Sparse Guidance Network [18.415854443539786]
  We propose a camera-based semantic scene completion framework called SGN. SGN propagates semantics from semantic-aware seed voxels to the whole scene based on spatial geometry cues. Our experimental results demonstrate the superiority of our SGN over existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T04:17:27Z)
• Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding [11.416392706435415]
  Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language Models (VLMs) Existing strategies directly map Vision-Language Models from 2D pixels of rendered or captured views to 3D points, overlooking the inherent and expressible point cloud geometric structure. We introduce the first training-free aggregation technique that leverages the point cloud's 3D geometric structure to improve the quality of the transferred Vision-Language Models.
  arXiv  Detail & Related papers  (2023-12-04T12:30:07Z)
• Sphere2Vec: A General-Purpose Location Representation Learning over a Spherical Surface for Large-Scale Geospatial Predictions [73.60788465154572]
  Current 2D and 3D location encoders are designed to model point distances in Euclidean space. We propose a multi-scale location encoder called Sphere2Vec which can preserve spherical distances when encoding point coordinates on a spherical surface.
  arXiv  Detail & Related papers  (2023-06-30T12:55:02Z)
• PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking? [62.997667081978825]
  We encode 3D detections as nodes in a graph, where spatial and temporal pairwise relations among objects are encoded via localized polar coordinates on graph edges. This allows our graph neural network to learn to effectively encode temporal and spatial interactions. We establish a new state-of-the-art on nuScenes dataset and, more importantly, show that our method, PolarMOT, generalizes remarkably well across different locations.
  arXiv  Detail & Related papers  (2022-08-03T10:06:56Z)
• Concentric Spherical GNN for 3D Representation Learning [53.45704095146161]
  We propose a novel multi-resolution convolutional architecture for learning over concentric spherical feature maps. Our hierarchical architecture is based on alternatively learning to incorporate both intra-sphere and inter-sphere information. We demonstrate the effectiveness of our approach in improving state-of-the-art performance on 3D classification tasks with rotated data.
  arXiv  Detail & Related papers  (2021-03-18T19:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.