PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World

• URL: http://arxiv.org/abs/2605.13169v2
• Date: Fri, 15 May 2026 16:50:42 GMT
• Title: PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World
• Abstract summary: We study pano-native understanding, which requires an MLLM to reason over an equirectangular projection panorama as a continuous, observer-centered space.<n>We introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream.<n>Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks.
• Score: 20.19893789145407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Related papers

• PanoWorld: Real-World Panoramic Generation [68.3569360188378]
  We propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory.<n>Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.
  arXiv  Detail & Related papers  (2026-07-10T17:59:32Z)
• Pano2World: End-to-End 3D Generation via Unified Multi-View Sequences [9.89397063309918]
  We present Pano2World, which takes a single indoor panorama as input and directly outputs a persistent, explorable 3D Gaussian scene.<n> Experiments demonstrate that Pano2World significantly outperforms existing methods on the multi-position panoramic novel-view synthesis benchmark.
  arXiv  Detail & Related papers  (2026-07-01T11:54:02Z)
• OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning [94.47284498900895]
  We introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints.<n>For training, OmniCoT-T is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence.
  arXiv  Detail & Related papers  (2026-06-29T14:38:20Z)
• PanoWorld: Geometry-Consistent Panoramic Video World Modeling [31.274777055274345]
  We present PanoWorld, a panoramic video world model that generates geometry-consistent 360$degree$ video from a single image and a caption.<n> Experiments show that PanoWorld improves geometric consistency over prior panoramic generation methods while maintaining competitive visual realism.
  arXiv  Detail & Related papers  (2026-05-14T20:24:23Z)
• S3KF: Spherical State-Space Kalman Filtering for Panoramic 3D Multi-Object Tracking [24.79542959759503]
  Panoramic multi-object tracking is important for industrial safety monitoring, wide-area robotic perception, and infrastructure-light deployment in large workspaces.<n>Existing image-plane trackers are tightly coupled to the camera projection and become unreliable in panoramic imagery.<n>We present a panoramic 3D multi-object tracking framework built on a motorized rotating LiDAR and a quad-fisheye camera rig.
  arXiv  Detail & Related papers  (2026-03-29T06:01:29Z)
• PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery [41.23812453847979]
  PanoVGGT jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass.<n>In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations.
  arXiv  Detail & Related papers  (2026-03-18T10:24:06Z)
• More than the Sum: Panorama-Language Models for Adverse Omni-Scenes [63.9800546079941]
  Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding.<n>We introduce the Panorama-Language Modeling (PLM)paradigm, a unified vision-language reasoning that is more than the sum of its pinhole counterparts.
  arXiv  Detail & Related papers  (2026-03-10T12:19:50Z)
• PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion [87.13016347332943]
  PanoWorld-X is a novel framework for high-fidelity and controllable panoramic video generation with diverse camera trajectories.<n>Our experiments demonstrate superior performance in various aspects, including motion range, control precision, and visual quality.
  arXiv  Detail & Related papers  (2025-09-29T16:22:00Z)
• Dense360: Dense Understanding from Omnidirectional Panoramas [24.862817640267572]
  We introduce an omnidirectional panoramas dataset featuring a comprehensive suite of reliability-scored annotations.<n>Specifically, our dataset contains 160K panoramas with 5M dense entity-level captions, 1M unique referring expressions, and 100K entity-grounded panoramic scene descriptions.
  arXiv  Detail & Related papers  (2025-06-17T12:35:23Z)
• Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning? [66.88619941063048]
  We ask: Are multimodal large language models (MLLMs) ready for omnidirectional spatial reasoning?<n> OSR-Bench is the first benchmark specifically designed for this setting.<n>It includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps.<n>We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings.
  arXiv  Detail & Related papers  (2025-05-17T08:48:40Z)
• MCPDepth: Omnidirectional Depth Estimation via Stereo Matching from Multi-Cylindrical Panoramas [49.891712558113845]
  We introduce Multi-Cylindrical Panoramic Depth Estimation (MCPDepth)<n>MCPDepth is a two-stage framework designed to enhance omnidirectional depth estimation.<n>Our method improves the mean absolute error (MAE) by 18.8% on the outdoor dataset Deep360 and by 19.9% on the real dataset 3D60.
  arXiv  Detail & Related papers  (2024-08-03T03:35:37Z)
• PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas [54.4948540627471]
  We propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion. Results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods.
  arXiv  Detail & Related papers  (2023-06-02T13:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.