IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models

• URL: http://arxiv.org/abs/2602.16138v1
• Date: Wed, 18 Feb 2026 02:06:24 GMT
• Title: IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models
• Abstract summary: We introduce IRIS, a training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA.<n>Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs.<n>We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.
• Score: 2.3699325153255404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs, more than doubling the accuracy of responses on ambiguous questions (from 35.2% to 77.2%) while maintaining performance on unambiguous queries. We evaluate our approach across state-of-the-art VLMs, showing consistent improvements when gaze data is incorporated in ambiguous image-question pairs, regardless of architectural differences. We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.

Related papers

• NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities [18.746981067240387]
  We evaluate vision-language models (VLMs) on everyday visual questions on Xiaohongshu, a mainstream Chinese image-sharing platform.<n>We find that the highest short-answer accuracy is 52.8%, while adding agentic search to Qwen3.5397B-A17B improves accuracy by only 2.0%.<n>These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.
  arXiv  Detail & Related papers  (2026-09-14T15:01:10Z)
• CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA [33.876705366050935]
  Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants.<n>We introduce CRAG-MM-Diagnostics, a diagnostic benchmark with stage-wise data annotations that isolate 1) language-based visual grounding, 2) object identification, and 3) knowledge retrieval and reasoning.<n>Our results point to knowledge retrieval and reasoning as the primary bottleneck, but also highlight issues in the other parts of the KI-VQA pipeline.
  arXiv  Detail & Related papers  (2026-07-23T10:37:01Z)
• Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents [3.1626173943755975]
  Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts.<n>Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck.<n>Introspective and Interactive Visual Grounding (IVG) is a framework that combines spec-grounded introspection and view-grounded interaction.
  arXiv  Detail & Related papers  (2026-04-22T22:47:23Z)
• Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models [77.3748853516374]
  We propose a training-free, model-intrinsic grounding method for vision-language models.<n>We compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map.<n>We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule.
  arXiv  Detail & Related papers  (2026-04-09T16:51:42Z)
• What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models [10.883552856100684]
  We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities.<n>We find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries.
  arXiv  Detail & Related papers  (2026-01-07T02:33:03Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)
• Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios [69.00444996464662]
  We propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables vision-language models to reason using visual crops corresponding to relevant entities.<n>Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting.
  arXiv  Detail & Related papers  (2025-01-08T18:31:16Z)
• Right this way: Can VLMs Guide Us to See More to Answer Questions? [11.693356269848517]
  In question-answering scenarios, humans assess whether the available information is sufficient and seek additional information if necessary. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.
  arXiv  Detail & Related papers  (2024-11-01T06:43:54Z)
• Trust but Verify: Programmatic VLM Evaluation in the Wild [62.14071929143684]
  Programmatic VLM Evaluation (PROVE) is a new benchmarking paradigm for evaluating VLM responses to open-ended queries. We benchmark the helpfulness-truthfulness trade-offs of a range ofVLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.
  arXiv  Detail & Related papers  (2024-10-17T01:19:18Z)
• The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation [32.7348470366509]
  The goal of RSVQA is to answer a question formulated in natural language about a remote sensing image. The problem of language biases is often overlooked in the remote sensing community. The present work aims at highlighting the problem of language biases in RSVQA with a threefold analysis strategy.
  arXiv  Detail & Related papers  (2023-11-28T13:45:15Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models [58.17315970207874]
  We propose a zero-shot method for adapting generalisable visual-textual priors from arbitrary VLM to facilitate moment-text alignment. Experiments conducted on three VMR benchmark datasets demonstrate the notable performance advantages of our zero-shot algorithm.
  arXiv  Detail & Related papers  (2023-09-01T13:06:50Z)
• Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering [6.798129852396113]
  In this paper, we explore effective prompting techniques to enhance zero- and few-shot Visual Question Answering (VQA) performance.<n>We identify that specific templates significantly influence VQA outcomes, underscoring the need for strategic template selection.<n>To mitigate the challenges associated with evaluating free-form open-ended VQA responses, we introduce a straightforward LLM-guided pre-processing technique.
  arXiv  Detail & Related papers  (2023-06-16T17:47:57Z)
• Contrast and Classify: Training Robust VQA Models [60.80627814762071]
  We propose a novel training paradigm (ConClaT) that optimize both cross-entropy and contrastive losses. We find that optimizing both losses -- either alternately or jointly -- is key to effective training.
  arXiv  Detail & Related papers  (2020-10-13T00:23:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.