Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs

• URL: http://arxiv.org/abs/2410.08145v1
• Date: Thu, 10 Oct 2024 17:31:17 GMT
• Title: Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
• Authors: Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Pinjia He, Zhaopeng Tu,
• Abstract summary: This paper explores the problem of commonsense-level vision-knowledge conflict in Multimodal Large Language Models (MLLMs) We introduce an automated pipeline, augmented with human-in-the-loop quality control, to establish a benchmark aimed at simulating and assessing the conflicts in MLLMs. We evaluate the conflict-resolution capabilities of nine representative MLLMs across various model families and find a noticeable over-reliance on textual queries.
• Score: 55.74117540987519
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper explores the problem of commonsense-level vision-knowledge conflict in Multimodal Large Language Models (MLLMs), where visual information contradicts model's internal commonsense knowledge (see Figure 1). To study this issue, we introduce an automated pipeline, augmented with human-in-the-loop quality control, to establish a benchmark aimed at simulating and assessing the conflicts in MLLMs. Utilizing this pipeline, we have crafted a diagnostic benchmark comprising 374 original images and 1,122 high-quality question-answer (QA) pairs. This benchmark covers two types of conflict target and three question difficulty levels, providing a thorough assessment tool. Through this benchmark, we evaluate the conflict-resolution capabilities of nine representative MLLMs across various model families and find a noticeable over-reliance on textual queries. Drawing on these findings, we propose a novel prompting strategy, "Focus-on-Vision" (FoV), which markedly enhances MLLMs' ability to favor visual data over conflicting textual knowledge. Our detailed analysis and the newly proposed strategy significantly advance the understanding and mitigating of vision-knowledge conflicts in MLLMs. The data and code are made publicly available.

Related papers

• Is Cognition consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding [15.828455477224516]
  As a multimodal task, document understanding requires models to possess both perceptual and cognitive abilities. In this paper, we define the conflicts between cognition and perception as Cognition and Perception (C&P) knowledge conflicts. We propose a novel method called Multimodal Knowledge Consistency Fine-tuning to mitigate the C&P knowledge conflicts.
  arXiv  Detail & Related papers  (2024-11-12T11:28:50Z)
• The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio [118.75449542080746]
  This paper presents the first systematic investigation of hallucinations in large multimodal models (LMMs) Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning.
  arXiv  Detail & Related papers  (2024-10-16T17:59:02Z)
• Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark [53.61633384281524]
  PolyMATH is a benchmark aimed at evaluating the general cognitive reasoning abilities of MLLMs. The best scores achieved on PolyMATH are 41%, 36%, and 27%, obtained by Claude-3.5 Sonnet, GPT-4o and Gemini-1.5 Pro respectively. A further fine-grained error analysis reveals that these models struggle to understand spatial relations and perform drawn-out, high-level reasoning.
  arXiv  Detail & Related papers  (2024-10-06T20:35:41Z)
• ECon: On the Detection and Resolution of Evidence Conflicts [56.89209046429291]
  The rise of large language models (LLMs) has significantly influenced the quality of information in decision-making systems. This study introduces a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios.
  arXiv  Detail & Related papers  (2024-10-05T07:41:17Z)
• Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models [33.76903352835436]
  Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. These models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components. We present a systematic approach to detect, interpret, and mitigate them.
  arXiv  Detail & Related papers  (2024-10-04T17:59:28Z)
• ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM [36.332500824079844]
  Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts has rarely been studied. We present ConflictBank, the first comprehensive benchmark developed to evaluate knowledge conflicts from three aspects. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences.
  arXiv  Detail & Related papers  (2024-08-22T02:33:13Z)
• MMRel: A Relation Understanding Benchmark in the MLLM Era [72.95901753186227]
  Multi-Modal Relation Understanding (MMRel) is a benchmark that features large-scale, high-quality, and diverse data on inter-object relations. MMRel is ideal for evaluating MLLMs on relation understanding, as well as for fine-tuning MLLMs to enhance relation comprehension capability.
  arXiv  Detail & Related papers  (2024-06-13T13:51:59Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Knowledge Conflicts for LLMs: A Survey [24.731074825915833]
  Survey focuses on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict. These conflicts can significantly impact the trustworthiness and performance of large language models.
  arXiv  Detail & Related papers  (2024-03-13T08:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.