Which Source Wins? Task-Dependent Reliance in Vision-Language Models

• URL: http://arxiv.org/abs/2608.17205v1
• Date: Mon, 17 Aug 2026 23:38:14 GMT
• Title: Which Source Wins? Task-Dependent Reliance in Vision-Language Models
• Abstract summary: Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them.<n>We study this modality reallocation with a controlled setup.<n>We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers.<n>We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin.
• Score: 0.6015914760581824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model's preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at https://github.com/Ro-netizen004/multimodal-arbitration-artifact.

Related papers

• VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification [53.25222629244709]
  VDiff-Bench is a benchmark for fine-grained Image Difference Identification.<n>Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor.<n>Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle.
  arXiv  Detail & Related papers  (2026-09-05T20:20:24Z)
• Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark [2.379911867541422]
  Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image.<n>They often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself.<n>Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question.
  arXiv  Detail & Related papers  (2026-06-09T04:18:38Z)
• VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning? [79.44137632338062]
  When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence?<n>We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question pairs across cardinality, attribute, spatial, and logic tasks.<n>We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images.
  arXiv  Detail & Related papers  (2026-06-05T22:06:07Z)
• Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs [23.958966900531692]
  Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens.<n>We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes.
  arXiv  Detail & Related papers  (2026-03-10T02:14:23Z)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges [85.24983823102262]
  We propose a structured methodology for evaluating text-to-image (T2I) models and vision language models (VLMs)<n>We test whether VLMs can identify 27 specific failure modes in the images generated by T2I models conditioned on challenging prompts.<n>Our findings suggest that current metrics are insufficient to capture these nuanced errors.
  arXiv  Detail & Related papers  (2025-12-01T19:46:03Z)
• RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning [63.599057862999]
  RefChartQA is a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding.<n>Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%.
  arXiv  Detail & Related papers  (2025-03-29T15:50:08Z)
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models [58.21452697997078]
  We propose a novel VQAttack model, which can generate both image and text perturbations with the designed modules. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQAttack.
  arXiv  Detail & Related papers  (2024-02-16T21:17:42Z)
• Vector-quantized Image Modeling with Improved VQGAN [93.8443646643864]
  We propose a Vector-quantized Image Modeling approach that involves pretraining a Transformer to predict image tokens autoregressively. We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. When trained on ImageNet at 256x256 resolution, we achieve Inception Score (IS) of 175.1 and Frechet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN.
  arXiv  Detail & Related papers  (2021-10-09T18:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.