What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

• URL: http://arxiv.org/abs/2607.10240v1
• Date: Sat, 11 Jul 2026 10:01:29 GMT
• Title: What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks
• Abstract summary: Short-answer VQA benchmarks conflate two distinct quantities.<n>We use a human-validated semantic judge to audit over 37k official errors.
• Score: 25.519850997497283
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Short-answer VQA benchmarks conflate two distinct quantities: whether a model's answer is semantically correct, and whether that answer matches the surface form expected by the automatic evaluator. We study this conflation across six vision--language models and six benchmarks, using a human-validated semantic judge (97.6% precision) to audit over 37k official errors. A second text-only judge reproduces the same benchmark-level false-negative pattern, showing that the effect is not an artifact of a single audit model. On text-rich benchmarks, up to half of these errors are semantically acceptable answers penalized purely for surface-form mismatch. This instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further destabilize official outcomes, flipping item-level correctness at substantial rates without changing the underlying task. A deterministic CPU-only contract repair confirms that the undercount is partially recoverable. These findings imply that official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable.

Related papers

• From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy [2.2917707112773593]
  Website redundancy does not have a single fixed meaning.<n>We introduce CORA, which measures repetition load, normal-use tax, and failure-domain recovery reserve separately.
  arXiv  Detail & Related papers  (2026-08-21T07:17:35Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge [0.0]
  We present our solution for Track 2 of the CVPR 2026 VidLLMs Challenge.<n>This track evaluates visual reasoning in videos, where models must infer relations that are not always explicitly visible.<n>We propose ASC, a training-free test-time reasoning framework built on a multimodal reasoning model.
  arXiv  Detail & Related papers  (2026-06-03T00:51:55Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• Rethinking Atomic Decomposition for LLM Judges: A Prompt-Controlled Study of Reference-Grounded QA Evaluation [5.94231111588812]
  We compare a self-decomposing atomic judge (single-prompt decompose-and-verify) against a prompt-controlled holistic judge with the same inputs and a similarly detailed rubric.<n>Our finding is specific to the self-decomposing single-prompt pattern on three QA-style benchmarks with 200 source examples each.
  arXiv  Detail & Related papers  (2026-03-30T03:55:26Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Gaming the Answer Matcher: Examining the Impact of Text Manipulation on Automated Judgment [6.104512852467398]
  Automated answer matching shows substantial promise as a scalable and aligned alternative to human evaluation.<n>We investigate whether such tactics deceive answer matching models by prompting examinee models to generate verbose responses.<n>Our results show that these manipulations do not increase scores and often reduce them.
  arXiv  Detail & Related papers  (2025-12-22T17:39:13Z)
• Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries [85.81295563405433]
  We present a protocol that synthetically constructs context surrounding an under-specified query and provides it during evaluation.<n>We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts.
  arXiv  Detail & Related papers  (2024-11-11T18:58:38Z)
• Measuring short-form factuality in large language models [50.15055025275888]
  We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. SimpleQA is adversarially collected against GPT-4 responses. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted.
  arXiv  Detail & Related papers  (2024-11-07T01:58:42Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [74.11403803488643]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.<n>We show that QASemConsistency yields factual consistency scores that correlate well with human judgments.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• A Semantic-based Method for Unsupervised Commonsense Question Answering [40.18557352036813]
  Unsupervised commonsense question answering is appealing since it does not rely on any labeled task data. We present a novel SEmantic-based Question Answering method (SEQA) for unsupervised commonsense question answering.
  arXiv  Detail & Related papers  (2021-05-31T08:21:52Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.