Fully Authentic Visual Question Answering Dataset from Online Communities

• URL: http://arxiv.org/abs/2311.15562v4
• Date: Wed, 17 Jul 2024 07:28:19 GMT
• Title: Fully Authentic Visual Question Answering Dataset from Online Communities
• Authors: Chongyan Chen, Mengchen Liu, Noel Codella, Yunsheng Li, Lu Yuan, Danna Gurari,
• Abstract summary: Visual Question Answering (VQA) entails answering questions about images. We introduce the first VQA dataset in which all contents originate from an authentic use case. We characterize this dataset and how it relates to eight mainstream VQA datasets.
• Score: 72.0524198499719
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Question Answering (VQA) entails answering questions about images. We introduce the first VQA dataset in which all contents originate from an authentic use case. Sourced from online question answering community forums, we call it VQAonline. We characterize this dataset and how it relates to eight mainstream VQA datasets. Observing that answers in our dataset tend to be much longer (i.e., a mean of 173 words) and so incompatible with standard VQA evaluation metrics, we instead utilize popular metrics for longer text evaluation for evaluating six state-of-the-art VQA models on VQAonline and report where they struggle most. Finally, we analyze which evaluation metrics align best with human judgments. To facilitate future extensions, we publicly-share the dataset at: https://vqaonline.github.io/.

Related papers

• Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts [54.072432123447854]
  Visual question answering (VQA) is the task of answering questions about an image. Answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts not present in the image. We propose a framework that uses language guidance (LG) in the form of rationales, image captions, scene graphs, etc to answer questions more accurately.
  arXiv  Detail & Related papers  (2023-10-31T03:54:11Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Making the V in Text-VQA Matter [1.2962828085662563]
  Text-based VQA aims at answering questions by reading the text present in the images. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context.
  arXiv  Detail & Related papers  (2023-08-01T05:28:13Z)
• BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models [47.64219291655723]
  We introduce a new test set for visual question answering (VQA) called BinaryVQA to push the limits of VQA models. Our dataset includes 7,800 questions across 1,024 images and covers a wide variety of objects, topics, and concepts. Around 63% of the questions have positive answers.
  arXiv  Detail & Related papers  (2023-01-28T00:03:44Z)
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [39.788346536244504]
  A-OKVQA is a crowdsourced dataset composed of about 25K questions. We demonstrate the potential of this new dataset through a detailed analysis of its contents.
  arXiv  Detail & Related papers  (2022-06-03T17:52:27Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)
• Grounding Answers for Visual Questions Asked by Visually Impaired People [16.978747012406266]
  VizWiz-VQA-Grounding is the first dataset that visually grounds answers to visual questions asked by people with visual impairments. We analyze our dataset and compare it with five VQA-Grounding datasets to demonstrate what makes it similar and different.
  arXiv  Detail & Related papers  (2022-02-04T06:47:16Z)
• Human-Adversarial Visual Question Answering [62.30715496829321]
  We benchmark state-of-the-art VQA models against human-adversarial examples. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples.
  arXiv  Detail & Related papers  (2021-06-04T06:25:32Z)
• 'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks [11.299897008333241]
  GQA is a dataset for real-world visual reasoning and compositional question answering. Many answers predicted by the best vision models on the GQA dataset do not match the ground-truth answer but still are semantically meaningful and correct in the given context. We propose Alternative Answer Sets (AAS) of ground-truth answers to address this limitation, which is created automatically using off-the-shelf NLP tools.
  arXiv  Detail & Related papers  (2021-03-28T00:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.