Accounting for Focus Ambiguity in Visual Questions

• URL: http://arxiv.org/abs/2501.02201v1
• Date: Sat, 04 Jan 2025 05:36:11 GMT
• Title: Accounting for Focus Ambiguity in Visual Questions
• Authors: Chongyan Chen, Yu-Yun Tseng, Zhuoheng Li, Anush Venkatesh, Danna Gurari,
• Abstract summary: We introduce VQ-FocusAmbiguity, the first VQA dataset that visually grounds each region described in the question that is necessary to arrive at the answer. We benchmark modern models for two novel tasks: recognizing whether a visual question has focus ambiguity and localizing all plausible focus regions within the image.
• Score: 14.696389149373994
• License:
• Abstract: No existing work on visual question answering explicitly accounts for ambiguity regarding where the content described in the question is located in the image. To fill this gap, we introduce VQ-FocusAmbiguity, the first VQA dataset that visually grounds each region described in the question that is necessary to arrive at the answer. We then provide an analysis showing how our dataset for visually grounding `questions' is distinct from visually grounding `answers', and characterize the properties of the questions and segmentations provided in our dataset. Finally, we benchmark modern models for two novel tasks: recognizing whether a visual question has focus ambiguity and localizing all plausible focus regions within the image. Results show that the dataset is challenging for modern models. To facilitate future progress on these tasks, we publicly share the dataset with an evaluation server at https://focusambiguity.github.io/.

Related papers

• Ask Questions with Double Hints: Visual Question Generation with Answer-awareness and Region-reference [107.53380946417003]
  We propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. We develop a simple methodology to self-learn the visual hints without introducing any additional human annotations.
  arXiv  Detail & Related papers  (2024-07-06T15:07:32Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Making the V in Text-VQA Matter [1.2962828085662563]
  Text-based VQA aims at answering questions by reading the text present in the images. Recent studies have shown that the question-answer pairs in the dataset are more focused on the text present in the image. The models trained on this dataset predict biased answers due to the lack of understanding of visual context.
  arXiv  Detail & Related papers  (2023-08-01T05:28:13Z)
• What's Different between Visual Question Answering for Machine "Understanding" Versus for Accessibility? [8.373151777137792]
  In visual question answering (VQA), a machine must answer a question given an associated image. We evaluate discrepancies between machine "understanding" datasets (VQA-v2) and accessibility datasets (VizWiz) by evaluating a variety of VQA models. Based on our findings, we discuss opportunities and challenges in VQA for accessibility and suggest directions for future work.
  arXiv  Detail & Related papers  (2022-10-26T18:23:53Z)
• From Pixels to Objects: Cubic Visual Attention for Visual Question Answering [132.95819467484517]
  Recently, attention-based Visual Question Answering (VQA) has achieved great success by utilizing question to target different visual areas that are related to the answer. We propose a Cubic Visual Attention (CVA) model by successfully applying a novel channel and spatial attention on object regions to improve VQA task. Experimental results show that our proposed method significantly outperforms the state-of-the-arts.
  arXiv  Detail & Related papers  (2022-06-04T07:03:18Z)
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [39.788346536244504]
  A-OKVQA is a crowdsourced dataset composed of about 25K questions. We demonstrate the potential of this new dataset through a detailed analysis of its contents.
  arXiv  Detail & Related papers  (2022-06-03T17:52:27Z)
• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering [75.53187719777812]
  This paper revisits visual representation in knowledge-based visual question answering (VQA) We propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions. We achieve new state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous state-of-the-art method by a large margin.
  arXiv  Detail & Related papers  (2022-06-02T17:59:56Z)
• Grounding Answers for Visual Questions Asked by Visually Impaired People [16.978747012406266]
  VizWiz-VQA-Grounding is the first dataset that visually grounds answers to visual questions asked by people with visual impairments. We analyze our dataset and compare it with five VQA-Grounding datasets to demonstrate what makes it similar and different.
  arXiv  Detail & Related papers  (2022-02-04T06:47:16Z)
• Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering [71.6781118080461]
  We propose a Graph Matching Attention (GMA) network for Visual Question Answering (VQA) task. firstly, it builds graph for the image, but also constructs graph for the question in terms of both syntactic and embedding information. Next, we explore the intra-modality relationships by a dual-stage graph encoder and then present a bilateral cross-modality graph matching attention to infer the relationships between the image and the question. Experiments demonstrate that our network achieves state-of-the-art performance on the GQA dataset and the VQA 2.0 dataset.
  arXiv  Detail & Related papers  (2021-12-14T10:01:26Z)
• Point and Ask: Incorporating Pointing into Visual Question Answering [14.744503080484977]
  We introduce and motivate point-input questions as an extension of Visual Question Answering (VQA) Pointing is a nearly universal gesture among humans, and real-world VQA is likely to involve a gesture towards the target region. We uncover and address several visual recognition challenges, including the ability to infer human intent.
  arXiv  Detail & Related papers  (2020-11-27T11:43:45Z)
• Visual Question Answering on Image Sets [70.4472272672716]
  We introduce the task of Image-Set Visual Question Answering (ISVQA), which generalizes the commonly studied single-image VQA problem to multi-image settings. Taking a natural language question and a set of images as input, it aims to answer the question based on the content of the images. The questions can be about objects and relationships in one or more images or about the entire scene depicted by the image set.
  arXiv  Detail & Related papers  (2020-08-27T08:03:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.