OpenView: Empowering MLLMs with Out-of-view VQA

• URL: http://arxiv.org/abs/2512.18563v1
• Date: Sun, 21 Dec 2025 02:11:40 GMT
• Title: OpenView: Empowering MLLMs with Out-of-view VQA
• Abstract summary: This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason objects, activities, and scenes beyond the visible frame of a perspective view.<n>We design OpenView, a four-stage pipeline to massively generate multi-choice VQA by leveraging panoramic imagery.<n>We build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation.
• Score: 73.09076507585404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline to massively generate multi-choice VQA by leveraging panoramic imagery to enable context-rich and spatial-grounded VQA synthesis with free-view framing. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, upon empowered by OpenView, multiple MLLMs can consistently boost their performance, uplifted from 48.6% to 64.1% on average. Code, benchmark, and data will be available at https://github.com/q1xiangchen/OpenView.

Related papers

• CoV: Chain-of-View Prompting for Spatial Reasoning [64.77921266428824]
  Chain-of-View (CoV) prompting transforms a vision--language model into an active viewpoint reasoner.<n>We evaluate CoV on OpenEQA and obtain an average +11.56% improvement in LLM-Match, with a maximum gain of +13.62% on Qwen3-VL-Flash.
  arXiv  Detail & Related papers  (2026-01-08T17:59:42Z)
• OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesis [80.3346344429389]
  We introduce OmniView, a unified framework that generalizes across a wide range of 4D consistency tasks.<n>Our method separately represents space, time, and view conditions, enabling flexible combinations of these inputs.<n>For example, OmniView can synthesize novel views from static, dynamic, and multiview inputs, extrapolate trajectories forward and backward in time, and create videos from text or image prompts with full camera control.
  arXiv  Detail & Related papers  (2025-12-11T18:59:05Z)
• Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs [27.872566171986946]
  Large vision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality.<n>To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views.<n>We present E3VQA, the first benchmark for multi-view question answering with 4K high-quality question-answer pairs grounded in synchronized ego-exo image pairs.
  arXiv  Detail & Related papers  (2025-05-28T04:09:42Z)
• Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos [66.1935609072708]
  LangView is a framework that uses the relative accuracy of view-dependent caption predictions as a proxy for best view pseudo-labels.<n>During inference, our model takes as input only a multi-view video--no language or camera poses--and returns the best viewpoint to watch at each timestep.
  arXiv  Detail & Related papers  (2024-11-13T16:31:08Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension [62.40482764691584]
  We introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating textbftext-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs. We conduct a thorough evaluation involving 34 prominent MLLMs and emphasize the current limitations of MLLMs in text-rich visual comprehension.
  arXiv  Detail & Related papers  (2024-04-25T17:39:35Z)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning [22.440669015518015]
  We evaluate whether multi-modal large language models (MLLMs) possess abstract visual reasoning abilities. Similar to the Sudoku puzzles, abstract visual reasoning (AVR) problems require finding high-level patterns. We introduce MARVEL, a benchmark with 770 MLLMs composed of six core knowledge patterns, geometric and abstract shapes, and five different task configurations.
  arXiv  Detail & Related papers  (2024-04-21T09:15:02Z)
• POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World [59.545114016224254]
  Humans are good at translating third-person observations of hand-object interactions into an egocentric view. We propose a Prompt-Oriented View-agnostic learning framework, which enables this view adaptation with few egocentric videos.
  arXiv  Detail & Related papers  (2024-03-09T09:54:44Z)
• Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding [33.33424214458285]
  Vision language models (VLM) have demonstrated remarkable performance across various downstream tasks. However, understanding fine-grained visual-linguistic concepts, such as attributes and inter-object relationships, remains a significant challenge. We introduce a progressive pipeline to synthesize images that vary in a specific attribute while ensuring consistency in all other aspects.
  arXiv  Detail & Related papers  (2023-11-30T03:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.