OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning
- URL: http://arxiv.org/abs/2606.30378v1
- Date: Mon, 29 Jun 2026 14:38:20 GMT
- Title: OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning
- Abstract summary: We introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints.<n>For training, OmniCoT-T is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence.
- Score: 94.47284498900895
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal Large Language Models (MLLMs) have demonstrated promising spatial reasoning capabilities, while these abilities remain underexplored in the emerging visual modality of panoramic imagery. The full 360°$\times$180° field of view of panoramas essentially supports complex global multi-step reasoning, which is also the fundamental advantage of panoramas in applications such as embodied intelligence. However, existing panoramic benchmarks largely focus on simplistic queries that rely on local cues or single-/few-step reasoning, thereby ignoring the fundamental advantage of panoramas and failing to fully exploit their potential. To address this gap, we introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints. It includes OmniCoT-B (6.7K data) for evaluation, which measures both answer accuracy and reasoning quality, OmniCoT-Real (1K data) as a manually annotated real-world subset to quantify the Sim-to-Real gap. For training, OmniCoT-T (14.3K data) is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence. Based on OmniCoT-T, we introduce OmniCoT-R1 and adopt a two-stage training strategy tailored to the geometrically complex panoramic space, where Supervised Fine-tuning (SFT) anchors reasoning to panoramic evidence (e.g., bearings, proximity) and GRPO penalizes geometrically incoherent paths to consolidate global 360° spatial consistency. Through OmniCoT, we aim to recalibrate the difficulty of panoramic spatial reasoning to better align with the intrinsic capabilities of panoramic imagery, thereby fostering meaningful progress in this research area.
Related papers
- OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping [69.48762031013514]
We propose OmniView-Space, a framework designed to maintain spatial consistency through multimodal egocentric evidence.<n>Our approach consists of three core components: (1) Multi-Perspective Spatial Mapping (MPSM), which re-anchors reconstructed geometry into a query-aligned visual cognitive map and a textual spatial graph; (2) Tool-Guided Egocentric Reasoning, an interleaved policy trained to actively select the ego anchor required by the query and request the corresponding MPSM evidence; and (3) Cognitive-Map Distillation, which uses MPSM-generated trajectories and ego-frame rewards to train the model to reason with self-generated cognitive maps
arXiv Detail & Related papers (2026-07-01T12:45:12Z) - PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World [20.19893789145407]
We study pano-native understanding, which requires an MLLM to reason over an equirectangular projection panorama as a continuous, observer-centered space.<n>We introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream.<n>Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks.
arXiv Detail & Related papers (2026-05-13T08:31:22Z) - Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning [61.753025885751036]
We introduce the Ego-to-World benchmark, which evaluates a vision-language model's ability to fuse heterogeneous viewpoints across three tasks.<n>We propose CoRL, a two-stage framework that combines Chain-of-Thought supervised fine-tuning with reinforcement learning.<n>We show that CoRL consistently surpasses strong proprietary and open-source baselines on both reasoning and perception-grounding metrics.
arXiv Detail & Related papers (2026-03-16T04:27:53Z) - More than the Sum: Panorama-Language Models for Adverse Omni-Scenes [63.9800546079941]
Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding.<n>We introduce the Panorama-Language Modeling (PLM)paradigm, a unified vision-language reasoning that is more than the sum of its pinhole counterparts.
arXiv Detail & Related papers (2026-03-10T12:19:50Z) - Universal Pansharpening Foundation Model [67.10467574892282]
Pansharpening generates the high-resolution multi-spectral (MS) image by integrating spatial details from a texture-rich panchromatic (PAN) image and spectral attributes from a low-resolution MS image.<n>We present FoundPS, a universal pansharpening foundation model for satellite-agnostic and scene-robust fusion.
arXiv Detail & Related papers (2026-03-04T08:30:15Z) - PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion [87.13016347332943]
PanoWorld-X is a novel framework for high-fidelity and controllable panoramic video generation with diverse camera trajectories.<n>Our experiments demonstrate superior performance in various aspects, including motion range, control precision, and visual quality.
arXiv Detail & Related papers (2025-09-29T16:22:00Z) - Omnidirectional Spatial Modeling from Correlated Panoramas [4.75637997496421]
Existing omnidirectional methods achieve scene understanding within a single frame while neglecting cross-frame correlated panoramas.<n>We introduce textbfCFpano, the textbffirst benchmark dataset dedicated to cross-frame correlated panoramas visual question answering.<n>We present methodname, a multi-modal large language model (MLLM) fine-tuned with Group Relative Policy Optimization ( GRPO) and a set of tailored reward functions for robust and consistent reasoning with cross-frame correlated panoramas.
arXiv Detail & Related papers (2025-09-02T10:14:55Z) - Dense360: Dense Understanding from Omnidirectional Panoramas [24.862817640267572]
We introduce an omnidirectional panoramas dataset featuring a comprehensive suite of reliability-scored annotations.<n>Specifically, our dataset contains 160K panoramas with 5M dense entity-level captions, 1M unique referring expressions, and 100K entity-grounded panoramic scene descriptions.
arXiv Detail & Related papers (2025-06-17T12:35:23Z) - Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method [8.039453341761538]
We introduce OmniVQA, the first dataset and conduct the first benchmark for omnidirectional visual question answering.<n>Our evaluation of state-of-the-art MLLMs reveals significant limitations in handling omnidirectional visual question answering.<n>We introduce a rule-based reinforcement learning method, 360-R1, based on Qwen2.5-VL-Instruct.
arXiv Detail & Related papers (2025-05-20T10:55:26Z) - Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning? [66.88619941063048]
We ask: Are multimodal large language models (MLLMs) ready for omnidirectional spatial reasoning?<n> OSR-Bench is the first benchmark specifically designed for this setting.<n>It includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps.<n>We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings.
arXiv Detail & Related papers (2025-05-17T08:48:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.