Multimodal Prompt Retrieval for Generative Visual Question Answering

• URL: http://arxiv.org/abs/2306.17675v1
• Date: Fri, 30 Jun 2023 14:06:13 GMT
• Title: Multimodal Prompt Retrieval for Generative Visual Question Answering
• Abstract summary: We propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text. Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting.
• Score: 9.973591610073006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA). Despite the recent advances in VQA, existing methods mainly adopt a discriminative formulation that predicts answers within a pre-defined label set, leading to easy overfitting on low-resource domains with limited labeled data (e.g., medicine) and poor generalization under domain shift to another dataset. To tackle this limitation, we propose a novel generative model enhanced by multimodal prompt retrieval (MPR) that integrates retrieved prompts and multimodal features to generate answers in free text. Our generative model enables rapid zero-shot dataset adaptation to unseen data distributions and open-set answer labels across datasets. Our experiments on medical VQA tasks show that MPR outperforms its non-retrieval counterpart by up to 30% accuracy points in a few-shot domain adaptation setting.

Related papers

• Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs [2.0532572974867174]
  Gastrointestinal (GI) endoscopic image analysis has shifted from single-label classification toward visual question answering (VQA)<n>We propose a simple multi-task fine-tuning recipe that constructs auxiliary grounding and description tasks from an existing VQA dataset.<n>We show consistent accuracy gains together with improved implicit alignment between answer tokens and the relevant image region.
  arXiv  Detail & Related papers  (2026-07-29T16:49:19Z)
• Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation [6.370860526262166]
  Existing frameworks for active sample selection offer limited support for generative Question Answering tasks.<n>We present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging Large Language Models (LLM) as surrogates for informing the sample selection process.<n>Our zero-shot acquisition functions reduce estimation error by 40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.
  arXiv  Detail & Related papers  (2026-02-26T22:03:51Z)
• MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering [51.19392014547221]
  Retrieval-based multimodal document QA aims to identify and integrate relevant information from visually rich documents with complex multimodal structures.<n>Current approaches rely on query-agnostic document representations that overlook salient content.<n>We propose the Multimodal Adaptive Retrieval-Augmented (MARA) framework, which introduces query-adaptive generation.
  arXiv  Detail & Related papers  (2026-02-01T12:27:40Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation [72.34977512403643]
  Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus.<n>Existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images)<n>We propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for Universal Retrieval-Augmented Generation scenarios.
  arXiv  Detail & Related papers  (2025-10-20T09:56:43Z)
• VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples [40.65748922891177]
  VaccineRAG is a novel Chain-of-Thought-based retrieval-augmented generation dataset.<n> VaccineRAG employs a benchmark to evaluate models using data with varying positive/negative sample ratios.<n>It enhances models' sample-discrimination capabilities by prompting LLMs to generate explicit Chain-of-Thought analysis for each sample.
  arXiv  Detail & Related papers  (2025-09-02T04:49:51Z)
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning [5.1583788731239455]
  The scaling of large language models to encode all the world's knowledge is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. This paper introduces RAVEN, a retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T13:08:35Z)
• SEMQA: Semi-Extractive Multi-Source Question Answering [94.04430035121136]
  We introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. We create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions.
  arXiv  Detail & Related papers  (2023-11-08T18:46:32Z)
• Adapting Pre-trained Generative Models for Extractive Question Answering [4.993041970406846]
  We introduce a novel approach that uses the power of pre-trained generative models to address extractive QA tasks. We demonstrate the superior performance of our proposed approach compared to existing state-of-the-art models.
  arXiv  Detail & Related papers  (2023-11-06T09:01:02Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias [92.41919689753051]
  Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. We investigate training data generation with diversely attributed prompts, which have the potential to yield diverse and attributed generated data. We show that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance.
  arXiv  Detail & Related papers  (2023-06-28T03:31:31Z)
• Combining Data Generation and Active Learning for Low-Resource Question Answering [23.755283239897132]
  We propose a novel approach that combines data augmentation via question-answer generation with Active Learning to improve performance in low-resource settings. Our findings show that our novel approach, where humans are incorporated in a data generation approach, boosts performance in the low-resource, domain-specific setting.
  arXiv  Detail & Related papers  (2022-11-27T16:31:33Z)
• Intermediate Training on Question Answering Datasets Improves Generative Data Augmentation [32.83012699501051]
  We improve generative data augmentation by formulating the data generation as context generation task. We cast downstream tasks into question answering format and adapt the fine-tuned context generators to the target task domain. We demonstrate substantial improvements in performance in few-shot, zero-shot settings.
  arXiv  Detail & Related papers  (2022-05-25T09:28:21Z)
• Abstractive Query Focused Summarization with Query-Free Resources [60.468323530248945]
  In this work, we consider the problem of leveraging only generic summarization resources to build an abstractive QFS system. We propose Marge, a Masked ROUGE Regression framework composed of a novel unified representation for summaries and queries. Despite learning from minimal supervision, our system achieves state-of-the-art results in the distantly supervised setting.
  arXiv  Detail & Related papers  (2020-12-29T14:39:35Z)
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs [73.93607719921945]
  We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
  arXiv  Detail & Related papers  (2020-01-22T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.