A Comprehensive Survey on Visual Question Answering Datasets and Algorithms

• URL: http://arxiv.org/abs/2411.11150v1
• Date: Sun, 17 Nov 2024 18:52:06 GMT
• Title: A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
• Authors: Raihan Kabir, Naznin Haque, Md Saiful Islam, Marium-E-Jannat,
• Abstract summary: We meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We explore six main paradigms of VQA models: fusion, attention, the technique of using information from one modality to filter information from another, external knowledge base, composition or reasoning, and graph models.
• Score: 1.941892373913038
• License:
• Abstract: Visual question answering (VQA) refers to the problem where, given an image and a natural language question about the image, a correct natural language answer has to be generated. A VQA model has to demonstrate both the visual understanding of the image and the semantic understanding of the question, demonstrating reasoning capability. Since the inception of this field, a plethora of VQA datasets and models have been published. In this article, we meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We divide VQA datasets into four categories: (1) available datasets that contain a rich collection of authentic images, (2) synthetic datasets that contain only synthetic images produced through artificial means, (3) diagnostic datasets that are specially designed to test model performance in a particular area, e.g., understanding the scene text, and (4) KB (Knowledge-Based) datasets that are designed to measure a model's ability to utilize outside knowledge. Concurrently, we explore six main paradigms of VQA models: fusion, where we discuss different methods of fusing information between visual and textual modalities; attention, the technique of using information from one modality to filter information from another; external knowledge base, where we discuss different models utilizing outside information; composition or reasoning, where we analyze techniques to answer advanced questions that require complex reasoning steps; explanation, which is the process of generating visual and textual descriptions to verify sound reasoning; and graph models, which encode and manipulate relationships through nodes in a graph. We also discuss some miscellaneous topics, such as scene text understanding, counting, and bias reduction.

Related papers

• Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference [107.53380946417003]
  We propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. We develop a simple methodology to self-learn the visual hints without introducing any additional human annotations.
  arXiv  Detail & Related papers  (2024-07-06T15:07:32Z)
• ViCLEVR: A Visual Reasoning Dataset and Hybrid Multimodal Fusion Model for Visual Question Answering in Vietnamese [1.6340299456362617]
  We introduce the ViCLEVR dataset, a pioneering collection for evaluating various visual reasoning capabilities in Vietnamese. We conduct a comprehensive analysis of contemporary visual reasoning systems, offering valuable insights into their strengths and limitations. We present PhoVIT, a comprehensive multimodal fusion that identifies objects in images based on questions.
  arXiv  Detail & Related papers  (2023-10-27T10:44:50Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Making the V in Text-VQA Matter [1.2962828085662563]
  Text-based VQA aims at answering questions by reading the text present in the images. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context.
  arXiv  Detail & Related papers  (2023-08-01T05:28:13Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [39.788346536244504]
  A-OKVQA is a crowdsourced dataset composed of about 25K questions. We demonstrate the potential of this new dataset through a detailed analysis of its contents.
  arXiv  Detail & Related papers  (2022-06-03T17:52:27Z)
• MGA-VQA: Multi-Granularity Alignment for Visual Question Answering [75.55108621064726]
  Learning to answer visual questions is a challenging task since the multi-modal inputs are within two feature spaces. We propose Multi-Granularity Alignment architecture for Visual Question Answering task (MGA-VQA) Our model splits alignment into different levels to achieve learning better correlations without needing additional data and annotations.
  arXiv  Detail & Related papers  (2022-01-25T22:30:54Z)
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding [140.5911760063681]
  We propose a novel dataset named Knowledge-Routed Visual Question Reasoning for VQA model evaluation. We generate the question-answer pair based on both the Visual Genome scene graph and an external knowledge base with controlled programs.
  arXiv  Detail & Related papers  (2020-12-14T00:33:44Z)
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering [27.042604046441426]
  Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. In this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. We decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA.
  arXiv  Detail & Related papers  (2020-08-31T23:25:01Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.