Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems

• URL: http://arxiv.org/abs/2604.14799v1
• Date: Thu, 16 Apr 2026 09:23:22 GMT
• Title: Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
• Abstract summary: Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
• Score: 52.83669998269706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective abstention (EA), recognizing evidence insufficiency and refraining from answering, is critical for reliable multimodal systems. Yet existing evaluation paradigms for vision-language models (VLMs) and multi-agent systems (MAS) assume answerability, pushing models to always respond. Abstention has been studied in text-only settings but remains underexplored multimodally; current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes. We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes: visual modality dependency and evidence sufficiency. Evaluating three frontier VLMs spanning closed and open-source models and two MAS architectures across 2079 samples, we find: (1) under standard prompting, VLMs rarely abstain; even simple confidence baselines outperform this setup, (2) MAS improves abstention but introduces an accuracy-abstention trade-off, (3) sequential designs match or exceed iterative variants, suggesting the bottleneck is miscalibration rather than reasoning depth, and (4) models abstain when image or text evidence is absent, but attempt reconciliation with degraded or contradictory evidence. Effective multimodal abstention requires abstention-aware training rather than better prompting or more agents.

Related papers

• Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs [46.40763822929566]
  We study whether thinking and non-thinking interfaces preserve acceptable final-response behavior.<n>PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning.<n>We develop textbfPatternRM, a response-level reward model, and textbfPatternRL, which introduces pattern-specific penalties during reinforcement learning.
  arXiv  Detail & Related papers  (2026-08-13T03:44:45Z)
• When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding [14.452663025714811]
  We study absent answer detection for multimodal large language models (MLLMs) in video understanding.<n>We find that MLLMs overwhelmingly select plausible distractors rather than detecting the absent answer.<n>These findings expose a systematic failure in absent answer detection and highlight the need for explicit detection mechanisms in multimodal systems.
  arXiv  Detail & Related papers  (2026-06-06T15:51:25Z)
• LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations [53.20772659095155]
  This paper tackles the more challenging setting of IML under training-time incomplete observations.<n>We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task.
  arXiv  Detail & Related papers  (2026-05-01T06:11:42Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Benchmarking Deflection and Hallucination in Large Vision-Language Models [25.176271096443482]
  Existing benchmarks overlook conflicts between visual and textual evidence.<n>We introduce VLM-DeflectionBench, a benchmark of 2,775 samples spanning diverse multimodal retrieval settings.<n>Our results highlight the need to evaluate not only what models know, but how they behave when they do not.
  arXiv  Detail & Related papers  (2026-04-13T20:22:22Z)
• PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning [0.0]
  We study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain.<n>We propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning.<n>Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning.
  arXiv  Detail & Related papers  (2026-04-06T09:54:16Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration [40.720288165545476]
  We introduce an enhanced diffusion model as a pluggable mid-stage training module to effectively restore missing features.<n>Our strategy introduces two key innovations: (I) Dynamic Modality Gating, which adaptively leverages conditional features to steer the generation of semantically consistent features; (II) Cross-Modal Mutual Learning mechanism, which bridges the semantic spaces of dual encoders to achieve bidirectional alignment.
  arXiv  Detail & Related papers  (2026-02-03T06:06:35Z)
• When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models [75.16145284285456]
  We introduce VLA-Fool, a comprehensive study of multimodal adversarial robustness in embodied VLA models under both white-box and black-box settings.<n>We develop the first automatically crafted and semantically guided prompting framework.<n> Experiments on the LIBERO benchmark reveal that even minor multimodal perturbations can cause significant behavioral deviations.
  arXiv  Detail & Related papers  (2025-11-20T10:14:32Z)
• Think Twice: Branch-and-Rethink Reasoning Reward Model [32.70732791642558]
  We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling.<n>We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks.<n>By converting all-at-oncescoringintofocused, second-lookreasoning, BR-RMreducesjudgmentdiffusionandimproves sensitivity to subtle yet consequential errors.
  arXiv  Detail & Related papers  (2025-10-27T17:58:07Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.