Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

• URL: http://arxiv.org/abs/2609.05540v1
• Date: Wed, 02 Sep 2026 11:52:02 GMT
• Title: Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models
• Abstract summary: ASD diagnosis requires behavioral and developmental evidence, not static facial photographs.<n>We introduce PARITY, a synthetic, demographically balanced set of identity-controlled neutral/expression portrait pairs with neutral-neutral controls.<n>We find a clear split between refusal-first models and speculative models; in the latter, certain expressions disproportionately trigger harmful selections.
• Score: 35.534692995586035
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Many medical conditions require diagnosis through detailed, multi-context clinical assessment rather than from visual appearance alone. Despite this, vision-language models (VLMs) are increasingly queried to interpret images in ways that touch on medical or diagnostic judgments, raising safety concerns when such inferences are unsupported. ASD diagnosis requires behavioral and developmental evidence, not static facial photographs. We audit whether VLMs abstain from this unanswerable paired-image query, and whether expressions sway non-abstaining choices. We introduce PARITY (Paired Assessment with Reused Identity), a synthetic, demographically balanced set of identity-controlled neutral/expression portrait pairs with neutral-neutral controls. All identities are synthetic and have no ASD status; because the query is unanswerable from images, any non-abstaining selection is treated as a harmful attribution. Across contemporary VLMs, we find a clear split between refusal-first models and speculative models; in the latter, certain expressions disproportionately trigger harmful selections. Clinical guardrails and single-image framing substantially increase abstention, suggesting actionable mitigations in both prompting and interface design

Related papers

• Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming [41.40761036883243]
  Vision-Language Models (VLMs) have significantly advanced medical visual question answering, yet their performance in ultrasound remains suboptimal.<n>We propose a framework specifically designed to consider the sonographer's cognitive workflow.<n>We first introduce a structured Zoom-then-Diagnose paradigm, which replicates the interactive search process to enable lesion-focused reasoning.
  arXiv  Detail & Related papers  (2026-05-20T19:06:34Z)
• Medical Context Distorts Decisions in Clinical Vision Language Models [7.537887356414114]
  Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios remains poorly characterized.<n>This paper identifies three failure modes: modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs.
  arXiv  Detail & Related papers  (2026-05-17T13:11:38Z)
• MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage [20.835664121303534]
  Vision Language Models (VLMs) are increasingly used for tasks like medical report generation and visual question answering.<n>In clinical practice, interpretation begins with pre-diagnostic sanity checks.<n>Existing benchmarks largely assume this step is solved, and therefore miss a critical failure mode.<n>We introduce MedObvious, a 1,880-task benchmark that isolates input validation as a set-level consistency capability.
  arXiv  Detail & Related papers  (2026-03-24T17:59:54Z)
• When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models [0.5461938536945722]
  Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior.<n>We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images.
  arXiv  Detail & Related papers  (2026-03-04T12:33:36Z)
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models [54.48710348910535]
  Existing medical reasoning benchmarks primarily focus on analyzing a patient's condition based on an image from a single visit.<n>We introduce TemMed-Bench, the first benchmark designed for analyzing changes in patients' conditions between different clinical visits.
  arXiv  Detail & Related papers  (2025-09-29T17:51:26Z)
• DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis? [1.1094764204428438]
  We propose DrVD-Bench, the first benchmark for clinical visual reasoning.<n>DrVD-Bench consists of three modules: Visual Evidence, Reasoning Trajectory Assessment, and Report Generation Evaluation.<n>Our benchmark covers 20 task types, 17 diagnostic categories, and five imaging modalities-CT, MRI, ultrasound, radiography, and pathology.
  arXiv  Detail & Related papers  (2025-05-30T03:33:25Z)
• Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models [57.58426038241812]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks.<n>These models still suffer from hallucinations when required to implicitly recognize or infer diverse visual entities from images.<n>We propose a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks.
  arXiv  Detail & Related papers  (2024-12-29T23:56:01Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• GraVIS: Grouping Augmented Views from Independent Sources for Dermatology Analysis [52.04899592688968]
  We propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks.
  arXiv  Detail & Related papers  (2023-01-11T11:38:37Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.