NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

• URL: http://arxiv.org/abs/2609.15695v1
• Date: Mon, 14 Sep 2026 15:01:10 GMT
• Title: NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities
• Abstract summary: We evaluate vision-language models (VLMs) on everyday visual questions on Xiaohongshu, a mainstream Chinese image-sharing platform.<n>We find that the highest short-answer accuracy is 52.8%, while adding agentic search to Qwen3.5397B-A17B improves accuracy by only 2.0%.<n>These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.
• Score: 18.746981067240387
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users on Xiaohongshu, a mainstream Chinese image-sharing platform, continue to turn to other people for help with everyday visual questions. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 10 frontier VLMs, the highest short-answer accuracy is 52.8\%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0\%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.

Related papers

• Companion-style QA Assistance in Ego-Vision [55.53073147495282]
  We introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video.<n>It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks.<n>We propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism.
  arXiv  Detail & Related papers  (2026-09-06T16:49:45Z)
• Self-Evolving Visual Questioner [53.949845616431624]
  Vision-language models (VLMs) are typically trained as passive answerers.<n>We show that a VLM can continuously improve itself as a visual questioner without any external supervision.<n>We propose a self-evolving framework that uses a VLM itself as both a proposer and a filter to produce harder, more informative, and visual-centric questions.
  arXiv  Detail & Related papers  (2026-06-11T21:45:46Z)
• IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models [2.3699325153255404]
  We introduce IRIS, a training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA.<n>Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs.<n>We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.
  arXiv  Detail & Related papers  (2026-02-18T02:06:24Z)
• WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios [19.156760664417718]
  We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering capabilities of multi-model AI assistant on wearable devices like smart glasses.<n>WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry.<n>The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains.
  arXiv  Detail & Related papers  (2025-11-27T06:44:49Z)
• Respond Beyond Language: A Benchmark for Video Generation in Response to Realistic User Intents [30.228721661677493]
  RealVideoQuest is designed to evaluate the abilities of text-to-video (T2V) models in answering real-world, visually grounded queries.<n>It identifies 7.5K real user queries with video response intents and builds 4.5K high-quality query-video pairs.<n>Experiments indicate that current T2V models struggle with effectively addressing real user queries.
  arXiv  Detail & Related papers  (2025-06-02T13:52:21Z)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models [59.0256377330646]
  Lens is a benchmark with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios.<n>This dataset intrinsically supports to evaluate MLLMs to handle image-invariable prompts, from basic perception to compositional reasoning.<n>We evaluate 15+ frontier MLLMs such as Qwen2.5-VL-72B, InternVL3-78B, GPT-4o and two reasoning models QVQ-72B-preview and Kimi-VL.
  arXiv  Detail & Related papers  (2025-05-21T15:06:59Z)
• Seeking and Updating with Live Visual Knowledge [75.25025869244837]
  We introduce LiveVQA, the first-of-its-kind dataset featuring 107,143 samples and 12 categories data.<n>LiveVQA enables evaluation of how models handle latest visual information beyond their knowledge boundaries.<n>Our comprehensive benchmarking of 17 state-of-the-art MLLMs reveals significant performance gaps on content beyond knowledge cutoff.
  arXiv  Detail & Related papers  (2025-04-07T17:39:31Z)
• VQA$^2$: Visual Question Answering for Video Quality Assessment [76.81110038738699]
  Video Quality Assessment (VQA) is a classic field in low-level visual perception.<n>Recent studies in the image domain have demonstrated that Visual Question Answering (VQA) can enhance markedly low-level visual quality evaluation.<n>We introduce the VQA2 Instruction dataset - the first visual question answering instruction dataset that focuses on video quality assessment.<n>The VQA2 series models interleave visual and motion tokens to enhance the perception of spatial-temporal quality details in videos.
  arXiv  Detail & Related papers  (2024-11-06T09:39:52Z)
• An Evaluation of GPT-4V and Gemini in Online VQA [31.77015255871848]
  We evaluate two state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering dataset. We conduct fine-grained analysis by generating seven types of metadata for nearly 2,000 visual questions. Our zero-shot performance analysis highlights the types of questions that are most challenging for both models.
  arXiv  Detail & Related papers  (2023-12-17T07:38:43Z)
• Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering [47.668572102657684]
  This work introduces a novel, explainable multi-agent collaboration framework by leveraging the expansive knowledge of Large Language Models (LLMs) to enhance the capabilities of Vision Language Models (VLMs)
  arXiv  Detail & Related papers  (2023-11-29T03:10:42Z)
• ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese [1.6340299456362617]
  We introduce the ViCLEVR dataset, a pioneering collection for evaluating various visual reasoning capabilities in Vietnamese. We conduct a comprehensive analysis of contemporary visual reasoning systems, offering valuable insights into their strengths and limitations. We present PhoVIT, a comprehensive multimodal fusion that identifies objects in images based on questions.
  arXiv  Detail & Related papers  (2023-10-27T10:44:50Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Learning to Answer Questions in Dynamic Audio-Visual Scenarios [81.19017026999218]
  We focus on the Audio-Visual Questioning (AVQA) task, which aims to answer questions regarding different visual objects sounds, and their associations in videos. Our dataset contains more than 45K question-answer pairs spanning over different modalities and question types. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-SIC, V-SIC, and AVQA approaches.
  arXiv  Detail & Related papers  (2022-03-26T13:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.