VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

• URL: http://arxiv.org/abs/2606.07872v1
• Date: Fri, 05 Jun 2026 22:06:07 GMT
• Title: VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
• Abstract summary: When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence?<n>We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question pairs across cardinality, attribute, spatial, and logic tasks.<n>We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images.
• Score: 79.44137632338062
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/

Related papers

• Mirage Probes: How Vision Models Fake Visual Understanding [13.299409525312491]
  Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided.<n>We show that mirage behavior is linearly decodable from internal activations across residual stream, post-attention, and attention-head sites.
  arXiv  Detail & Related papers  (2026-06-11T19:51:44Z)
• Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving [12.905518450115906]
  We introduce a multi-view visual question answering benchmark for evaluating evidence-source identification.<n>The benchmark contains 122 conflict-centric question-answer pairs from 73 scenes.<n>By explicitly separating visual-source identification from answer correctness, the benchmark exposes grounding failures that answer-only evaluation misses.
  arXiv  Detail & Related papers  (2026-06-08T15:39:06Z)
• Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)? [72.2500547961037]
  spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments.<n>We introduce two types of observation challenges: (1) occlusion, which hides target information, and (2) perspective ambiguity, which produces misleading visual cues.<n>For each configuration, we design spatial questions that are answerable under clean observations but require abstention under the introduced challenges.
  arXiv  Detail & Related papers  (2026-05-28T20:44:47Z)
• Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination [39.713386491199884]
  Vision-Language Models (VLMs) often produce self-reflective statements like "let me check the figure again" during reasoning.<n>Do such statements trigger genuine visual re-examination, or are they merely learned textual patterns?<n>We investigate this via VisualSwap, an image-swap probing framework.
  arXiv  Detail & Related papers  (2026-05-15T11:31:14Z)
• Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts [7.071615105094556]
  We show that visual attributes can be linearly decodable from early layers.<n>We show that training-free activation in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups.
  arXiv  Detail & Related papers  (2026-04-10T14:36:07Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs [18.349695067647012]
  Visual Language Models excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple tests.<n>We present an evaluation that tests vision-language models' capacity for nonlocal visual reasoning.<n>Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-04T23:15:52Z)
• v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning [27.688428439248607]
  We introduce v1, a lightweight extension that enables active visual referencing through a simple point-and-copy approach.<n>This allows the model to identify relevant image patches and copy their embeddings back into the reasoning stream.<n>Our pointing strategy lets the MLLM directly select image patches using their semantic representations as keys, keeping perceptual evidence embedded in the same space as the model's reasoning.
  arXiv  Detail & Related papers  (2025-05-24T19:30:47Z)
• MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
  We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
  arXiv  Detail & Related papers  (2025-05-01T17:41:49Z)
• Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions [103.20281438405111]
  Multiple-choice question answering (MCQA) is a key competence of performant transformer language models.<n>We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer.<n>We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles.
  arXiv  Detail & Related papers  (2024-07-21T00:10:23Z)
• Counterfactual Samples Synthesizing for Robust Visual Question Answering [104.72828511083519]
  We propose a model-agnostic Counterfactual Samples Synthesizing (CSS) training scheme. CSS generates numerous counterfactual training samples by masking critical objects in images or words in questions. We achieve a record-breaking performance of 58.95% on VQA-CP v2, with 6.5% gains.
  arXiv  Detail & Related papers  (2020-03-14T08:34:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.