More than the Sum: Panorama-Language Models for Adverse Omni-Scenes

• URL: http://arxiv.org/abs/2603.09573v1
• Date: Tue, 10 Mar 2026 12:19:50 GMT
• Title: More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
• Abstract summary: Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding.<n>We introduce the Panorama-Language Modeling (PLM)paradigm, a unified vision-language reasoning that is more than the sum of its pinhole counterparts.
• Score: 63.9800546079941
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-Language Modeling (PLM)paradigm, a unified $360^\circ$ vision-language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA dataset that involves adverse omni-scenes, enabling comprehensive reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.

Related papers

• Pano2World: End-to-End 3D Generation via Unified Multi-View Sequences [9.89397063309918]
  We present Pano2World, which takes a single indoor panorama as input and directly outputs a persistent, explorable 3D Gaussian scene.<n> Experiments demonstrate that Pano2World significantly outperforms existing methods on the multi-position panoramic novel-view synthesis benchmark.
  arXiv  Detail & Related papers  (2026-07-01T11:54:02Z)
• OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning [94.47284498900895]
  We introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints.<n>For training, OmniCoT-T is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence.
  arXiv  Detail & Related papers  (2026-06-29T14:38:20Z)
• PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World [20.19893789145407]
  We study pano-native understanding, which requires an MLLM to reason over an equirectangular projection panorama as a continuous, observer-centered space.<n>We introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream.<n>Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T08:31:22Z)
• PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms [39.219645333154794]
  We introduce PanORama, the first panoptic segmentation for the operating room that is multiview-consistent by design.<n>By modeling cross-view interactions at the feature level inside the backbone in a single forward pass, view consistency emerges directly rather than through post-hoc refinement.<n>PanORama is calibration-free, requiring no camera parameters, and generalizes to unseen camera viewpoints within any multiview configuration at inference time.
  arXiv  Detail & Related papers  (2026-03-20T13:04:07Z)
• PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion [87.13016347332943]
  PanoWorld-X is a novel framework for high-fidelity and controllable panoramic video generation with diverse camera trajectories.<n>Our experiments demonstrate superior performance in various aspects, including motion range, control precision, and visual quality.
  arXiv  Detail & Related papers  (2025-09-29T16:22:00Z)
• Omnidirectional Spatial Modeling from Correlated Panoramas [4.75637997496421]
  Existing omnidirectional methods achieve scene understanding within a single frame while neglecting cross-frame correlated panoramas.<n>We introduce textbfCFpano, the textbffirst benchmark dataset dedicated to cross-frame correlated panoramas visual question answering.<n>We present methodname, a multi-modal large language model (MLLM) fine-tuned with Group Relative Policy Optimization ( GRPO) and a set of tailored reward functions for robust and consistent reasoning with cross-frame correlated panoramas.
  arXiv  Detail & Related papers  (2025-09-02T10:14:55Z)
• Dense360: Dense Understanding from Omnidirectional Panoramas [24.862817640267572]
  We introduce an omnidirectional panoramas dataset featuring a comprehensive suite of reliability-scored annotations.<n>Specifically, our dataset contains 160K panoramas with 5M dense entity-level captions, 1M unique referring expressions, and 100K entity-grounded panoramic scene descriptions.
  arXiv  Detail & Related papers  (2025-06-17T12:35:23Z)
• Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning? [66.88619941063048]
  We ask: Are multimodal large language models (MLLMs) ready for omnidirectional spatial reasoning?<n> OSR-Bench is the first benchmark specifically designed for this setting.<n>It includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps.<n>We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings.
  arXiv  Detail & Related papers  (2025-05-17T08:48:40Z)
• DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion [60.45000652592418]
  We propose a novel text-driven panoramic generation framework, DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. We show that DiffPano can generate consistent, diverse panoramic images with given unseen text descriptions and camera poses.
  arXiv  Detail & Related papers  (2024-10-31T17:57:02Z)
• PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas [54.4948540627471]
  We propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion. Results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods.
  arXiv  Detail & Related papers  (2023-06-02T13:35:07Z)
• Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning [97.37544023666833]
  We introduce panoramic panoptic segmentation as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to the agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
  arXiv  Detail & Related papers  (2021-03-01T09:37:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.