Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA

• URL: http://arxiv.org/abs/2406.12746v5
• Date: Wed, 09 Oct 2024 16:04:39 GMT
• Title: Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA
• Abstract summary: We propose Diversification, Evidence Truncation, and Combination for Knowledge-based Elucidation (DietCoke) DietCoke comprises of three stages: diversification, rationalization, and ensemble. Experiments show that DietCoke significantly outperforms state-of-the-art LLM-based baselines.
• Score: 8.498145119681437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge-based Visual Question-answering (K-VQA) often requires the use of background knowledge beyond the image. However, we discover that a single knowledge generation strategy is often insufficient for all K-VQA questions. To this end, we propose Diversification, Evidence Truncation, and Combination for Knowledge-based Elucidation (DietCoke), which utilizes a bundle of complementary question-answering tactics and aggregates their answers using textual rationales. DietCoke comprises of three stages: diversification, rationalization, and ensemble. The diversification stage generates three distinctive decision contexts, each leading to its own answer candidate. The rationalization stage generates two rationales, the automatic rationale and the mechanistic rationale, for each answer candidate using decorrelated techniques. Finally, in the ensemble stage, an LLM informed by the rationales selects one answer from the three candidates. Experiments show that DietCoke significantly outperforms state-of-the-art LLM-based baselines by 2.8% on OK-VOA and 4.7% on A-OKVOA and that the strategies in the ensembles are highly complementary. Code is available at: https://github.com/limiaoyu/DietCoke

Related papers

• Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification [10.412378789389544]
  We propose CoA, an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed.<n>CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer.<n>CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines.
  arXiv  Detail & Related papers  (2026-01-23T02:12:33Z)
• Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage [74.70255719194819]
  We introduce a novel framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We use this framework to evaluate three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. We find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions.
  arXiv  Detail & Related papers  (2024-10-20T22:59:34Z)
• Causal Reasoning through Two Layers of Cognition for Improving Generalization in Visual Question Answering [28.071906755200043]
  Generalization in Visual Question Answering (VQA) requires models to answer questions about images with contexts beyond the training distribution. We propose Cognitive pathways VQA (CopVQA) improving the multimodal predictions by emphasizing causal reasoning factors. CopVQA achieves a new state-of-the-art (SOTA) on PathVQA dataset and comparable accuracy to the current SOTA on VQA-CPv2, VQAv2, and VQA RAD, with one-fourth of the model size.
  arXiv  Detail & Related papers  (2023-10-09T05:07:58Z)
• Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering [83.74210749046551]
  We propose to leverage question decomposing for heterogeneous knowledge integration. We propose a novel two-stage XQA framework, Reasoning over Hierarchical Question Decomposition Tree (RoHT) Experiments on complex QA datasets KQA Pro and Musique show that our framework outperforms SOTA methods significantly.
  arXiv  Detail & Related papers  (2023-05-24T11:45:59Z)
• Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources [87.26486246513063]
  Chain-of-knowledge (CoK) is a framework that augments large language models. CoK consists of three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation.
  arXiv  Detail & Related papers  (2023-05-22T17:34:23Z)
• Query Enhanced Knowledge-Intensive Conversation via Unsupervised Joint Modeling [35.27735234588822]
  We propose an unsupervised query enhanced approach for knowledge-intensive conversations, namely QKConv. QKConv is optimized through joint training, which produces the response by exploring multiple candidate queries and leveraging corresponding selected knowledge.
  arXiv  Detail & Related papers  (2022-12-19T16:21:05Z)
• Multifaceted Improvements for Conversational Open-Domain Question Answering [54.913313912927045]
  We propose a framework with Multifaceted Improvements for Conversational open-domain Question Answering (MICQA) Firstly, the proposed KL-divergence based regularization is able to lead to a better question understanding for retrieval and answer reading. Second, the added post-ranker module can push more relevant passages to the top placements and be selected for reader with a two-aspect constrains. Third, the well designed curriculum learning strategy effectively narrows the gap between the golden passage settings of training and inference, and encourages the reader to find true answer without the golden passage assistance.
  arXiv  Detail & Related papers  (2022-04-01T07:54:27Z)
• MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering [23.628740943735167]
  We propose MuKEA to represent multimodal knowledge by an explicit triplet to correlate visual objects and fact answers with implicit relations. By adopting a pre-training and fine-tuning learning strategy, both basic and domain-specific multimodal knowledge are progressively accumulated for answer prediction.
  arXiv  Detail & Related papers  (2022-03-17T07:42:14Z)
• ArT: All-round Thinker for Unsupervised Commonsense Question-Answering [54.068032948300655]
  We propose an approach of All-round Thinker (ArT) by fully taking association during knowledge generating. We evaluate it on three commonsense QA benchmarks: COPA, SocialIQA and SCT.
  arXiv  Detail & Related papers  (2021-12-26T18:06:44Z)
• Differentiable Open-Ended Commonsense Reasoning [80.94997942571838]
  We study open-ended commonsense reasoning (OpenCSR) using as a resource only a corpus of commonsense facts written in natural language. As an approach to OpenCSR, we propose DrFact, an efficient Differentiable model for multi-hop Reasoning over knowledge Facts.
  arXiv  Detail & Related papers  (2020-10-24T10:07:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.