A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning

• URL: http://arxiv.org/abs/2403.14972v2
• Date: Tue, 6 Aug 2024 09:45:27 GMT
• Title: A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning
• Authors: Changmeng Zheng, Dayong Liang, Wengyu Zhang, Xiao-Yong Wei, Tat-Seng Chua, Qing Li,
• Abstract summary: The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images. To address the issue, we propose a deductive (top-down) debating approach called Blueprint Debate on Graphs (BDoG) In BDoG, debates are confined to a blueprint graph to prevent opinion trivialization through world-level summarization. Moreover, by storing evidence in branches within the graph, BDoG mitigates distractions caused by frequent but irrelevant concepts.
• Score: 53.35861580821777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a pilot study aimed at introducing multi-agent debate into multimodal reasoning. The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images. These challenges stem from the inductive (bottom-up) nature of existing debating schemes. To address the issue, we propose a deductive (top-down) debating approach called Blueprint Debate on Graphs (BDoG). In BDoG, debates are confined to a blueprint graph to prevent opinion trivialization through world-level summarization. Moreover, by storing evidence in branches within the graph, BDoG mitigates distractions caused by frequent but irrelevant concepts. Extensive experiments validate that BDoG is able to achieve state-of-the-art results in ScienceQA and MMBench with significant improvements over previous methods. The source code can be accessed at https://github.com/thecharm/BDoG.

Related papers

• Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models [48.43678591317425]
  Boosting of Thoughts (BoT) is an automated prompting framework for problem solving with Large Language Models. We show that BoT consistently achieves higher or comparable problem-solving rates than other advanced prompting approaches.
  arXiv  Detail & Related papers  (2024-02-17T00:13:36Z)
• DebateKG: Automatic Policy Debate Case Creation with Semantic Knowledge Graphs [0.0]
  We show that effective debate cases can be constructed using constrained shortest path traversals on Argumentative Semantic Knowledge Graphs. We significantly improve upon DebateSum by introducing 53180 new examples. We create a unique method for evaluating which knowledge graphs are better in the context of producing policy debate cases.
  arXiv  Detail & Related papers  (2023-07-09T04:19:19Z)
• Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings [61.04460792203266]
  We introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks.
  arXiv  Detail & Related papers  (2023-05-03T17:58:29Z)
• Explaining Image Classification with Visual Debates [26.76139301708958]
  We propose a novel debate framework for understanding and explaining a continuous image classifier's reasoning for making a particular prediction. Our framework encourages players to put forward diverse arguments during the debates, picking up the reasoning trails missed by their opponents. We demonstrate and evaluate (a practical realization) our Visual Debates on the geometric SHAPE and MNIST datasets.
  arXiv  Detail & Related papers  (2022-10-17T12:35:52Z)
• CLEAR: Generative Counterfactual Explanations on Graphs [60.30009215290265]
  We study the problem of counterfactual explanation generation on graphs. A few studies have explored counterfactual explanations on graphs, but many challenges of this problem are still not well-addressed. We propose a novel framework CLEAR which aims to generate counterfactual explanations on graphs for graph-level prediction models.
  arXiv  Detail & Related papers  (2022-10-16T04:35:32Z)
• Deep Image Deblurring: A Survey [165.32391279761006]
  Deblurring is a classic problem in low-level computer vision, which aims to recover a sharp image from a blurred input image. Recent advances in deep learning have led to significant progress in solving this problem.
  arXiv  Detail & Related papers  (2022-01-26T01:31:30Z)
• Jointly Attacking Graph Neural Network and its Explanations [50.231829335996814]
  Graph Neural Networks (GNNs) have boosted the performance for many graph-related tasks. Recent studies have shown that GNNs are highly vulnerable to adversarial attacks, where adversaries can mislead the GNNs' prediction by modifying graphs. We propose a novel attack framework (GEAttack) which can attack both a GNN model and its explanations by simultaneously exploiting their vulnerabilities.
  arXiv  Detail & Related papers  (2021-08-07T07:44:33Z)
• On Generating Plausible Counterfactual and Semi-Factual Explanations for Deep Learning [15.965337956587373]
  PlausIble Exceptionality-based Contrastive Explanations (PIECE), modifies all exceptional features in a test image to be normal from the perspective of the counterfactual class. Two controlled experiments compare PIECE to others in the literature, showing that PIECE not only generates the most plausible counterfactuals on several measures, but also the best semifactuals.
  arXiv  Detail & Related papers  (2020-09-10T14:48:12Z)
• Debate Dynamics for Human-comprehensible Fact-checking on Knowledge Graphs [27.225048123690243]
  We propose a novel method for fact-checking on knowledge graphs based on debate dynamics. The underlying idea is to frame the task of triple classification as a debate game between two reinforcement learning agents. Our method allows for interactive reasoning on knowledge graphs where the users can raise additional arguments or evaluate the debate taking common sense reasoning and external information into account.
  arXiv  Detail & Related papers  (2020-01-09T15:19:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.