How to find a good image-text embedding for remote sensing visual question answering?

• URL: http://arxiv.org/abs/2109.11848v1
• Date: Fri, 24 Sep 2021 09:48:28 GMT
• Title: How to find a good image-text embedding for remote sensing visual question answering?
• Authors: Christel Chappuis, Sylvain Lobry, Benjamin Kellenberger, Bertrand Le Saux, Devis Tuia
• Abstract summary: Visual question answering (VQA) has been introduced to remote sensing to make information extraction from overhead imagery more accessible to everyone. We study three different fusion methodologies in the context of VQA for remote sensing and analyse the gains in accuracy with respect to the model complexity.
• Score: 41.0510495281302
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Visual question answering (VQA) has recently been introduced to remote sensing to make information extraction from overhead imagery more accessible to everyone. VQA considers a question (in natural language, therefore easy to formulate) about an image and aims at providing an answer through a model based on computer vision and natural language processing methods. As such, a VQA model needs to jointly consider visual and textual features, which is frequently done through a fusion step. In this work, we study three different fusion methodologies in the context of VQA for remote sensing and analyse the gains in accuracy with respect to the model complexity. Our findings indicate that more complex fusion mechanisms yield an improved performance, yet that seeking a trade-of between model complexity and performance is worthwhile in practice.

Related papers

• Large Vision-Language Models for Remote Sensing Visual Question Answering [0.0]
  Remote Sensing Visual Question Answering (RSVQA) is a challenging task that involves interpreting complex satellite imagery to answer natural language questions. Traditional approaches often rely on separate visual feature extractors and language processing models, which can be computationally intensive and limited in their ability to handle open-ended questions. We propose a novel method that leverages a generative Large Vision-Language Model (LVLM) to streamline the RSVQA process.
  arXiv  Detail & Related papers  (2024-11-16T18:32:38Z)
• Enhanced Textual Feature Extraction for Visual Question Answering: A Simple Convolutional Approach [2.744781070632757]
  We compare models that leverage long-range dependencies and simpler models focusing on local textual features within a well-established VQA framework. We propose ConvGRU, a model that incorporates convolutional layers to improve text feature representation without substantially increasing model complexity. Tested on the VQA-v2 dataset, ConvGRU demonstrates a modest yet consistent improvement over baselines for question types such as Number and Count.
  arXiv  Detail & Related papers  (2024-05-01T12:39:35Z)
• RTQ: Rethinking Video-language Understanding Based on Image-text Model [55.278942477715084]
  Video-language understanding presents unique challenges due to the inclusion of highly complex semantic details. We propose a novel framework called RTQ, which addresses these challenges simultaneously. Our model demonstrates outstanding performance even in the absence of video-language pre-training.
  arXiv  Detail & Related papers  (2023-12-01T04:51:01Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data [27.160303686163164]
  Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation. There are questions with clearly different difficulty levels for each image in the RSVQA task. A multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features.
  arXiv  Detail & Related papers  (2022-05-06T11:37:00Z)
• Achieving Human Parity on Visual Question Answering [67.22500027651509]
  The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image. This paper describes our recent research of AliceMind-MMU that obtains similar or even slightly better results than human beings does on VQA. This is achieved by systematically improving the VQA pipeline including: (1) pre-training with comprehensive visual and textual feature representation; (2) effective cross-modal interaction with learning to attend; and (3) A novel knowledge mining framework with specialized expert modules for the complex VQA task.
  arXiv  Detail & Related papers  (2021-11-17T04:25:11Z)
• Human-Adversarial Visual Question Answering [62.30715496829321]
  We benchmark state-of-the-art VQA models against human-adversarial examples. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples.
  arXiv  Detail & Related papers  (2021-06-04T06:25:32Z)
• Component Analysis for Visual Question Answering Architectures [10.56011196733086]
  The main goal of this paper is to provide a comprehensive analysis regarding the impact of each component in Visual Question Answering models. Our major contribution is to identify core components for training VQA models so as to maximize their predictive performance.
  arXiv  Detail & Related papers  (2020-02-12T17:25:50Z)
• Accuracy vs. Complexity: A Trade-off in Visual Question Answering Models [39.338304913058685]
  We study the trade-off between the model complexity and the performance on the Visual Question Answering task. We focus on the effect of "multi-modal fusion" in VQA models that is typically the most expensive step in a VQA pipeline.
  arXiv  Detail & Related papers  (2020-01-20T11:27:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.