Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images

• URL: http://arxiv.org/abs/2605.12413v3
• Date: Mon, 18 May 2026 10:53:40 GMT
• Title: Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
• Abstract summary: Multimodal Large Language Models (MLLMs) show strong visual perception, yet remain limited in reasoning about space under changing viewpoints.<n>We study this challenge as Perspective-Conditioned Spatial Reasoning (PCSR) in 360-degree omnidirectional images.<n>We introduce PCSR-Bench, a diagnostic benchmark of 84,373 question-answer pairs from 2,600 omnidirectional images.
• Score: 52.50752250573993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) show strong visual perception, yet remain limited in reasoning about space under changing viewpoints. We study this challenge as Perspective-Conditioned Spatial Reasoning (PCSR) in 360-degree omnidirectional images, where broad scene coverage reduces ambiguity from partial observations without eliminating the need for viewpoint-dependent inference. To assess this capability, we introduce PCSR-Bench, a diagnostic benchmark of 84,373 question-answer pairs from 2,600 omnidirectional images across 26 indoor environments. PCSR-Bench contains eight tasks spanning foundational perception (e.g., object counting, relative distance, and relative direction) and advanced PCSR, including compositional chains, egocentric rotation, perspective re-anchoring, ego-distortion, and limited-FOV visibility. We evaluate 14 representative MLLMs and observe a substantial perception-reasoning gap: accuracy reaches 57.59% on foundational relative direction, but drops to 13.49% on egocentric rotation, 7.13% on egocentric distortion, and 0.64% on open-ended compositional reasoning. To probe the plasticity of this gap, we conduct an RL-based diagnostic study on a 7B-scale model. Reward shaping improves a matched 7B baseline from 31.10% to 60.06% under a controlled setting, suggesting that PCSR is partial plasticity rather than being fully immutable. Still, the gains are task-selective, sensitive to reward design including both weight allocation and reward formulation, and partially dependent on the evaluation protocol. These results position PCSR as a key bottleneck in current MLLMs and highlight limited but meaningful room for recovery under targeted optimization.

Related papers

• SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward [31.36854122174582]
  Existing Vision-Language Models (VLMs) exhibits a critical bottleneck in robust spatial reasoning.<n>We propose SCOUT, a structured Chain-of-Thought framework that explicitly models 3D environmental perception.<n>We also introduce a novel RL algorithm featuring multi-objective process rewards and a tailored advantage estimation method.
  arXiv  Detail & Related papers  (2026-08-12T16:14:25Z)
• SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning [19.079142995255506]
  We propose SIVA-RL to replace operator-conditioned regularization with sample-wise, outcome-conditioned supervision.<n>SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image Patch.<n>It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.
  arXiv  Detail & Related papers  (2026-07-15T15:13:02Z)
• Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning [73.40085256022111]
  We propose Perceive-to-Reason (P2R), a unified framework that formulates fine-grained visual reasoning as a two-stage process.<n>P2R localizes question-relevant evidence as a Perceiver, and then answers the question as a Reasoner based on the annotated image and cropped regions.<n>PRA-GRPO is a role-aware reinforcement learning strategy that alternates between perception-focused and reasoning-focused updates.
  arXiv  Detail & Related papers  (2026-07-01T17:24:26Z)
• Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization [31.411469692692766]
  Chain-of-Thought traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence.<n>We propose Faithful GRPO, which enforces consistency and grounding as constraints via Lagrangian dual ascent.<n>Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%.
  arXiv  Detail & Related papers  (2026-04-09T17:15:47Z)
• Margin-Consistent Deep Subtyping of Invasive Lung Adenocarcinoma via Perturbation Fidelity in Whole-Slide Image Analysis [15.00702842825464]
  Whole-slide image classification for invasive lung adenocarcinoma subtyping remains vulnerable to real-world imaging perturbations.<n>We propose a margin consistency framework evaluated on 203,226 patches from 143 whole-slide images.<n>Our approach achieves robust feature-logit space alignment measured by Kendall correlations of 0.88 during training and 0.64 during validation.
  arXiv  Detail & Related papers  (2026-02-27T20:47:15Z)
• Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction [0.0]
  We propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates objectives into masked vision-language learning.<n>We evaluate Robust-MMR on multiple medical vision-language benchmarks, including medical visual question answering (VQA-RAD), cross-domain image-text classification (MELINDA), and robust image-caption retrieval (ROCO)<n>Our results show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical vision-language representations for real-world deployment.
  arXiv  Detail & Related papers  (2026-02-06T01:20:56Z)
• Token-Level Inference-Time Alignment for Vision-Language Models [58.41370989069588]
  Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence.<n>We present TITA, a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution.<n>During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback.
  arXiv  Detail & Related papers  (2025-10-20T09:58:03Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation [58.546205554954454]
  We propose Enhancing Alignment in MLLMs via Critical Observation (EACO)<n>EACO aligns MLLMs by self-generated preference data using only 5k images economically.<n>EACO reduces the overall hallucinations by 65.6% on HallusionBench and improves the reasoning ability by 21.8% on MME-Cognition.
  arXiv  Detail & Related papers  (2024-12-06T09:59:47Z)
• Sight View Constraint for Robust Point Cloud Registration [13.216523566864641]
  Partial to Partial Point Cloud Registration (partial PCR) is a challenging task, particularly when dealing with a low overlap rate. We propose a novel and general Sight View Constraint (SVC) to conclusively identify incorrect transformations. On the challenging 3DLoMatch dataset, our approach increases the registration recall from 78% to 82%, achieving the state-of-the-art result.
  arXiv  Detail & Related papers  (2024-09-08T11:58:20Z)
• LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference [66.45326873274908]
  We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%.
  arXiv  Detail & Related papers  (2023-07-23T03:38:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.