SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering

• URL: http://arxiv.org/abs/2509.20871v1
• Date: Thu, 25 Sep 2025 08:01:28 GMT
• Title: SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
• Authors: Yan Zhang, Jiaqing Lin, Miao Zhang, Kui Xiao, Xiaoju Hou, Yue Zhao, Zhifei Li,
• Abstract summary: We propose the Summarized Caption-Rerank Augmented VQA (SCRA-VQA)<n>SCRA-VQA employs a pre-trained visual language model to convert images into captions.<n>It generates contextual examples for the captions while simultaneously summarizing and reordering them to exclude unrelated information.
• Score: 15.985057987715974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Acquiring high-quality knowledge is a central focus in Knowledge-Based Visual Question Answering (KB-VQA). Recent methods use large language models (LLMs) as knowledge engines for answering. These methods generally employ image captions as visual text descriptions to assist LLMs in interpreting images. However, the captions frequently include excessive noise irrelevant to the question, and LLMs generally do not comprehend VQA tasks, limiting their reasoning capabilities. To address this issue, we propose the Summarized Caption-Rerank Augmented VQA (SCRA-VQA), which employs a pre-trained visual language model to convert images into captions. Moreover, SCRA-VQA generates contextual examples for the captions while simultaneously summarizing and reordering them to exclude unrelated information. The caption-rerank process enables LLMs to understand the image information and questions better, thus enhancing the model's reasoning ability and task adaptability without expensive end-to-end training. Based on an LLM with 6.7B parameters, SCRA-VQA performs excellently on two challenging knowledge-based VQA datasets: OK-VQA and A-OKVQA, achieving accuracies of 38.8% and 34.6%. Our code is available at https://github.com/HubuKG/SCRA-VQA.

Related papers

• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection [81.78173888579941]
  Large Language Models (LLMs) are considered a well-suited method to increase the quality of the question-answering functionality.<n>LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data.<n>This paper introduces a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question.
  arXiv  Detail & Related papers  (2025-07-18T12:28:08Z)
• GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance [0.9208007322096533]
  Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image.<n>Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources.<n>We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA)<n> Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information.
  arXiv  Detail & Related papers  (2025-05-25T23:00:30Z)
• Distill Visual Chart Reasoning Ability from LLMs to MLLMs [64.32993770646165]
  Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs)<n>We propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs.<n>ReachQA is a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities of MLLMs.
  arXiv  Detail & Related papers  (2024-10-24T14:50:42Z)
• Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts [3.6064695344878093]
  Visual question answering (VQA) is known as an AI-complete task as it requires understanding, reasoning, and inferring about the vision and the language content. This study explores the impact of incorporating image captioning as an intermediary process within the VQA pipeline.
  arXiv  Detail & Related papers  (2024-04-12T16:35:23Z)
• GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering [37.11794716736831]
  We argue that multimodal large language model (MLLM) is a better implicit knowledge engine than the large language model (LLM) for its superior capability of visual understanding. We propose GeReA, a generate-reason framework that prompts a MLLM like InstructBLIP with question relevant vision and language information to generate knowledge-relevant descriptions. Specifically, the question-relevant image regions and question-specific manual prompts are encoded in the MLLM to generate the knowledge relevant descriptions.
  arXiv  Detail & Related papers  (2024-02-04T14:28:23Z)
• Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions [15.262736501208467]
  Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge. As images are invisible to LLMs, researchers convert images to text to engage LLMs into the visual question reasoning procedure. We design a framework that enables LLMs to proactively ask relevant questions to unveil more details in the image.
  arXiv  Detail & Related papers  (2023-11-20T08:23:39Z)
• Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts [54.072432123447854]
  Visual question answering (VQA) is the task of answering questions about an image. Answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts not present in the image. We propose a framework that uses language guidance (LG) in the form of rationales, image captions, scene graphs, etc to answer questions more accurately.
  arXiv  Detail & Related papers  (2023-10-31T03:54:11Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models [111.42052290293965]
  Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. We propose emphImg2Prompt, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections.
  arXiv  Detail & Related papers  (2022-12-21T08:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.