Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

• URL: http://arxiv.org/abs/2403.17359v1
• Date: Tue, 26 Mar 2024 03:51:01 GMT
• Title: Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models
• Authors: Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu,
• Abstract summary: We present a Chain-of-Action framework for multimodal and retrieval-augmented Question-Answering (QA) Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information.
• Score: 17.60243337898751
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.

Related papers

• Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action [17.60243337898751]
  We present a Conversational Chain-of-Action (Conv-CoA) framework for Open-domain Conversational Question Answering (OCQA) Compared with literature, Conv-CoA addresses three major challenges: (i) unfaithful hallucination that is inconsistent with real-time or domain facts, (ii) weak reasoning performance in conversational scenarios, and (iii) unsatisfying performance in conversational information retrieval.
  arXiv  Detail & Related papers  (2024-05-28T04:46:52Z)
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting [124.69672273754144]
  Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs) Existing CoT approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. We introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts.
  arXiv  Detail & Related papers  (2024-03-21T11:34:26Z)
• Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering [55.295699268654545]
  We propose a novel Chain-of-Discussion framework to leverage the synergy among open-source Large Language Models. Our experiments show that discussions among multiple LLMs play a vital role in enhancing the quality of answers.
  arXiv  Detail & Related papers  (2024-02-26T05:31:34Z)
• Cross-Modal Reasoning with Event Correlation for Video Question Answering [32.332251488360185]
  We introduce the dense caption modality as a new auxiliary and distill event-correlated information from it to infer the correct answer. We employ cross-modal reasoning modules for explicitly modeling inter-modal relationships and aggregating relevant information across different modalities. We propose a question-guided self-adaptive multi-modal fusion module to collect the question-oriented and event-correlated evidence through multi-step reasoning.
  arXiv  Detail & Related papers  (2023-12-20T02:30:39Z)
• Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering [20.59485758381809]
  Current multimodal retrieval question-answering models face two main challenges. utilizing compressed evidence features as input to the model results in the loss of fine-grained information within the evidence. We propose a two-stage framework for evidence retrieval and question-answering to alleviate these issues.
  arXiv  Detail & Related papers  (2023-10-15T01:18:39Z)
• How Many Answers Should I Give? An Empirical Study of Multi-Answer Reading Comprehension [64.76737510530184]
  We design a taxonomy to categorize commonly-seen multi-answer MRC instances. We analyze how well different paradigms of current multi-answer MRC models deal with different types of multi-answer instances.
  arXiv  Detail & Related papers  (2023-06-01T08:22:21Z)
• Answering Questions by Meta-Reasoning over Multiple Chains of Thought [53.55653437903948]
  We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer.
  arXiv  Detail & Related papers  (2023-04-25T17:27:37Z)
• Multimodal Chain-of-Thought Reasoning in Language Models [94.70184390935661]
  We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2023-02-02T07:51:19Z)
• Multi-hop Inference for Question-driven Summarization [39.08269647808958]
  We propose a novel question-driven abstractive summarization method, Multi-hop Selective Generator (MSG) MSG incorporates multi-hop reasoning into question-driven summarization and, meanwhile, provide justifications for the generated summaries. Experimental results show that the proposed method consistently outperforms state-of-the-art methods on two non-factoid QA datasets.
  arXiv  Detail & Related papers  (2020-10-08T02:36:39Z)
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting [56.268862325167575]
  We tackle conversational passage retrieval (ConvPR) with query reformulation integrated into a multi-stage ad-hoc IR system. We propose two conversational query reformulation (CQR) methods: (1) term importance estimation and (2) neural query rewriting. For the former, we expand conversational queries using important terms extracted from the conversational context with frequency-based signals. For the latter, we reformulate conversational queries into natural, standalone, human-understandable queries with a pretrained sequence-tosequence model.
  arXiv  Detail & Related papers  (2020-05-05T14:30:20Z)
• Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering [40.58976291178477]
  We introduce a simple, fast, and unsupervised iterative evidence retrieval method. Despite its simplicity, our approach outperforms all the previous methods on the evidence selection task. When these evidence sentences are fed into a RoBERTa answer classification component, we achieve state-of-the-art QA performance.
  arXiv  Detail & Related papers  (2020-05-04T00:19:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.