What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

• URL: http://arxiv.org/abs/2601.06165v1
• Date: Wed, 07 Jan 2026 02:33:03 GMT
• Title: What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models
• Abstract summary: We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities.<n>We find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries.
• Score: 10.883552856100684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.

Related papers

• xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems [57.62172039072062]
  We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities.<n>The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements.<n>The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points.
  arXiv  Detail & Related papers  (2026-09-07T17:30:58Z)
• RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests [14.251266069918591]
  Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues.<n>We define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts.<n>We find that requests carrying only a problem statement account for 88% of real prompts but just 7% of benchmark problems.
  arXiv  Detail & Related papers  (2026-08-28T01:57:58Z)
• ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making [30.869819258358405]
  Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images.<n>We introduce COMPLEXITYWORLD, a benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories.
  arXiv  Detail & Related papers  (2026-08-05T10:05:41Z)
• VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild [0.0]
  LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying.<n>We attribute this gap to existing benchmarks' reliance on over-specified queries, single-turn interactions, and fixed-schema evaluation.<n>We term this paradigm VibeSearch and introduce VibeSearchBench, a benchmark comprising 200 manually curated bilingual (Chinese and English) tasks across 20 domains.
  arXiv  Detail & Related papers  (2026-05-27T03:06:18Z)
• ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues [6.879311587283136]
  We present ExPerT, a query-wise personalization framework that adapts large language models responses to users' query domain expertise.<n>ExPerT consists of two key components: (i) a semantic-behavioral expertise inference module that jointly interprets query text and keystroke dynamics.<n>Our user study with 40 participants and 1270 queries demonstrated that ExPerT reduced expertise inference error by 65.7% compared to the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-04T13:21:28Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• Seeing Through Words: Controlling Visual Retrieval Quality with Language Models [68.49490036960559]
  We propose a new paradigm of quality-controllable retrieval, which enriches short queries with contextual details while incorporating explicit notions of image quality.<n>Our key idea is to leverage a generative language model as a query completion function, extending underspecified queries into descriptive forms.<n>Our proposed approach significantly improves retrieval results and provides effective quality control, bridging the gap between the expressive capacity of modern VLMs and the underspecified nature of short user queries.
  arXiv  Detail & Related papers  (2026-02-24T18:20:57Z)
• IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models [2.3699325153255404]
  We introduce IRIS, a training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA.<n>Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs.<n>We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.
  arXiv  Detail & Related papers  (2026-02-18T02:06:24Z)
• WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts [14.966795545558474]
  This paper introduces WikiMixQA, a benchmark for evaluating cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages.<n>We evaluate 12 state-of-the-art vision-language models, revealing that while proprietary models achieve 70% accuracy when provided with direct context, their performance deteriorates significantly when retrieval from long documents is required.
  arXiv  Detail & Related papers  (2025-06-18T16:09:18Z)
• MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query [91.01970848241075]
  MERIT is the first multilingual dataset for interleaved multi-condition semantic retrieval.<n>This paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval.
  arXiv  Detail & Related papers  (2025-06-03T17:59:14Z)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models [48.99485386990197]
  We conduct a case study using a synthetic dataset solvable only through visual reasoning.<n>We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions.<n>Although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%.
  arXiv  Detail & Related papers  (2025-05-19T17:59:27Z)
• None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks [0.9831489366502301]
  We introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts.<n>Using this method, we evaluate state-of-the-art proprietary and open-source LLMs on two datasets available in English and Spanish.<n>Results show that all models experience remarkable accuracy drops, with an average loss of 57% on MMLU and 50% on UNED-Access 2024.
  arXiv  Detail & Related papers  (2025-02-18T14:32:44Z)
• Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries [85.81295563405433]
  We present a protocol that synthetically constructs context surrounding an under-specified query and provides it during evaluation.<n>We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts.
  arXiv  Detail & Related papers  (2024-11-11T18:58:38Z)
• On the Worst Prompt Performance of Large Language Models [93.13542053835542]
  Performance of large language models (LLMs) is acutely sensitive to the phrasing of prompts. We introduce RobustAlpacaEval, a new benchmark that consists of semantically equivalent case-level queries. Experiments on RobustAlpacaEval with ChatGPT and six open-source LLMs from the Llama, Mistral, and Gemma families uncover substantial variability in model performance.
  arXiv  Detail & Related papers  (2024-06-08T13:40:38Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.