SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.04244v1
• Date: Tue, 04 Aug 2026 21:55:47 GMT
• Title: SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models
• Abstract summary: Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues.<n>We introduce SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating text-vision conflict resolution.
• Score: 9.496555964908188
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing benchmarks rarely reveal how they arbitrate between these evidence sources when they conflict. We introduce SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating text-vision conflict resolution. Each source image is transformed into a counterfactual quintuplet of Original, Blank, Similar, Random, and Adversarial variants. Synthetic, localized scene-text interventions are designed to preserve non-textual content, enabling paired measurements of changes in localization performance and directed shifts toward geographic targets introduced by conflicting text. SIGNPOST-Bench contains 5,111 counterfactual groups and 25,555 image variants from four datasets. We evaluate 20 MLLMs from seven providers. Compared with Original images, Adversarial variants raise median localization error from 282 km to 1,347 km, a 4.8-fold increase. Among geocodable adversarial samples, 6.5-20.1% of predictions lie less than 50 km from the injected target across models, and every evaluated model exhibits a positive mean paired reduction in target distance from Blank to Adversarial. Compatible, unrelated, and conflicting text replacements produce distinct effects on model predictions, while clean-input localization performance does not fully predict robustness to conflicting text. These results establish visual geolocation as a continuous diagnostic of scene-text arbitration and provide a controlled framework for evaluating how MLLMs resolve conflicting multimodal evidence.

Related papers

• NumBench: Diagnosing Counting Failures in Text-to-Image Models [27.391694807526857]
  Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why.<n>We introduce textbf, a benchmark of 640,000 prompts spanning 1,600 categories and counts from 1 to 100.<n>We also develop a process model in which requested instances compete for a finite set of resolvable image regions.
  arXiv  Detail & Related papers  (2026-08-28T11:26:19Z)
• QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing [4.1436924146144944]
  Typographic attacks pose a critical threat to vision-language models.<n>QuISE is a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing.
  arXiv  Detail & Related papers  (2026-08-13T11:47:50Z)
• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• DisasterTD: Disaster Toponym Disambiguation Using Multimodal LLMs and Cross-View Geolocalization [38.854595481332275]
  Social media imagery (SMI) provides timely and fine-grained ground perspectives that are valuable for situational awareness and emergency response.<n> geographic references in SMI are often vague or ambiguous, making accurate geolocalization challenging.<n>We propose DisasterTD, a disaster toponym disambiguation framework that integrates multimodal large language model (MLLMs)-based semantic reasoning with cross-view geolocalization.
  arXiv  Detail & Related papers  (2026-07-26T02:51:53Z)
• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• Belief Propagation in LLM World Models: Measuring Strategic Information Bias with Prediction Markets [0.5999408856514375]
  We find that English news context systematically biases territorial predictions, wrong 64 to 72 percent of the time.<n>A contaminated model that knows actual outcomes shows the same error rate, indicating that the bias originates primarily in the text.<n>We show that the distortion originates primarily in the sources, not the models.
  arXiv  Detail & Related papers  (2026-05-12T09:22:45Z)
• Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs [23.958966900531692]
  Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens.<n>We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes.
  arXiv  Detail & Related papers  (2026-03-10T02:14:23Z)
• Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs [80.03370593724422]
  Out-of-distribution (OOD) detection seeks to identify samples from unknown classes.<n>Current methods often incorporate intra-modal distance during OOD detection, such as comparing negative texts with ID labels.<n>We propose InterNeg, a framework that systematically utilizes consistent inter-modal distance enhancement from textual and visual perspectives.
  arXiv  Detail & Related papers  (2026-03-03T05:44:47Z)
• Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs [17.56537230934894]
  Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks.<n>We show that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text.
  arXiv  Detail & Related papers  (2026-01-27T05:04:38Z)
• UniREditBench: A Unified Reasoning-based Image Editing Benchmark [52.54256348710893]
  This work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation.<n>It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions.<n>We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings.
  arXiv  Detail & Related papers  (2025-11-03T07:24:57Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict [3.9495397165148023]
  We show how vision-language models (VLMs) reason when confronted with scenarios that present conflicting image and text cues.<n>Our analysis shows that VLMs favor text in simpler queries but shift toward images as query complexity increases.<n>In addition, we explore three mitigation strategies: simple prompt modifications, modifications that explicitly instruct models on how to handle conflicting information.
  arXiv  Detail & Related papers  (2025-04-11T20:56:52Z)
• Less is More: Fewer Interpretable Region via Submodular Subset Selection [54.07758302264416]
  This paper re-models the above image attribution problem as a submodular subset selection problem. We construct a novel submodular function to discover more accurate small interpretation regions. For correctly predicted samples, the proposed method improves the Deletion and Insertion scores with an average of 4.9% and 2.5% gain relative to HSIC-Attribution.
  arXiv  Detail & Related papers  (2024-02-14T13:30:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.