Large Vision-Language Models for Remote Sensing Visual Question Answering

• URL: http://arxiv.org/abs/2411.10857v1
• Date: Sat, 16 Nov 2024 18:32:38 GMT
• Title: Large Vision-Language Models for Remote Sensing Visual Question Answering
• Authors: Surasakdi Siripong, Apirak Chaiyapan, Thanakorn Phonchai,
• Abstract summary: Remote Sensing Visual Question Answering (RSVQA) is a challenging task that involves interpreting complex satellite imagery to answer natural language questions. Traditional approaches often rely on separate visual feature extractors and language processing models, which can be computationally intensive and limited in their ability to handle open-ended questions. We propose a novel method that leverages a generative Large Vision-Language Model (LVLM) to streamline the RSVQA process.
• Score: 0.0
• License:
• Abstract: Remote Sensing Visual Question Answering (RSVQA) is a challenging task that involves interpreting complex satellite imagery to answer natural language questions. Traditional approaches often rely on separate visual feature extractors and language processing models, which can be computationally intensive and limited in their ability to handle open-ended questions. In this paper, we propose a novel method that leverages a generative Large Vision-Language Model (LVLM) to streamline the RSVQA process. Our approach consists of a two-step training strategy: domain-adaptive pretraining and prompt-based finetuning. This method enables the LVLM to generate natural language answers by conditioning on both visual and textual inputs, without the need for predefined answer categories. We evaluate our model on the RSVQAxBEN dataset, demonstrating superior performance compared to state-of-the-art baselines. Additionally, a human evaluation study shows that our method produces answers that are more accurate, relevant, and fluent. The results highlight the potential of generative LVLMs in advancing the field of remote sensing analysis.

Related papers

• Learning to Ground VLMs without Forgetting [54.033346088090674]
  We introduce LynX, a framework that equips pretrained Visual Language Models with visual grounding ability without forgetting their existing image and language understanding skills. To train the model effectively, we generate a high-quality synthetic dataset we call SCouT, which mimics human reasoning in visual grounding. We evaluate LynX on several object detection and visual grounding datasets, demonstrating strong performance in object detection, zero-shot localization and grounded reasoning.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets [9.67464173044675]
  Visual Question Answering (VQA) is the task of answering a question about an image. We present an approach for declarative knowledge distillation from Large Language Models (LLMs) Our results confirm that distilling knowledge from LLMs is in fact a promising direction besides data-driven rule learning approaches.
  arXiv  Detail & Related papers  (2024-10-12T08:17:03Z)
• Crafting Interpretable Embeddings by Asking LLMs Questions [89.49960984640363]
  Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli.
  arXiv  Detail & Related papers  (2024-05-26T22:30:29Z)
• The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation [32.7348470366509]
  The goal of RSVQA is to answer a question formulated in natural language about a remote sensing image. The problem of language biases is often overlooked in the remote sensing community. The present work aims at highlighting the problem of language biases in RSVQA with a threefold analysis strategy.
  arXiv  Detail & Related papers  (2023-11-28T13:45:15Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Tackling VQA with Pretrained Foundation Models without Further Training [0.0]
  Large language models (LLMs) have achieved state-of-the-art results in many natural language processing tasks. With the capability of these LLMs, researchers have looked into how to adopt them for use with Visual Question Answering (VQA) In this paper, we explore a method of combining pretrained LLMs and other foundation models without further training to solve the VQA problem.
  arXiv  Detail & Related papers  (2023-09-27T08:35:24Z)
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning [60.43585179885355]
  We propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. We conduct experiments on a range of knowledge-based visual reasoning datasets.
  arXiv  Detail & Related papers  (2023-01-12T18:59:50Z)
• From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data [27.160303686163164]
  Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation. There are questions with clearly different difficulty levels for each image in the RSVQA task. A multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features.
  arXiv  Detail & Related papers  (2022-05-06T11:37:00Z)
• How to find a good image-text embedding for remote sensing visual question answering? [41.0510495281302]
  Visual question answering (VQA) has been introduced to remote sensing to make information extraction from overhead imagery more accessible to everyone. We study three different fusion methodologies in the context of VQA for remote sensing and analyse the gains in accuracy with respect to the model complexity.
  arXiv  Detail & Related papers  (2021-09-24T09:48:28Z)
• Learning to Ask Conversational Questions by Optimizing Levenshtein Distance [83.53855889592734]
  We introduce a Reinforcement Iterative Sequence Editing (RISE) framework that optimize the minimum Levenshtein distance (MLD) through explicit editing actions. RISE is able to pay attention to tokens that are related to conversational characteristics. Experimental results on two benchmark datasets show that RISE significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-30T08:44:19Z)
• Visual Question Answering with Prior Class Semantics [50.845003775809836]
  We show how to exploit additional information pertaining to the semantics of candidate answers. We extend the answer prediction process with a regression objective in a semantic space. Our method brings improvements in consistency and accuracy over a range of question types.
  arXiv  Detail & Related papers  (2020-05-04T02:46:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.