COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes

• URL: http://arxiv.org/abs/2409.04053v1
• Date: Fri, 6 Sep 2024 06:49:55 GMT
• Title: COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes
• Authors: Koen Kraaijveld, Yifan Jiang, Kaixin Ma, Filip Ilievski,
• Abstract summary: We formulate visual lateral thinking as a multiple-choice question-answering task. We describe a three-step taxonomy-driven methodology for instantiating task examples. We develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles.
• Score: 14.603382370403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.

Related papers

• Help Me Identify: Is an LLM+VQA System All We Need to Identify Visual Concepts? [62.984473889987605]
  We present a zero-shot framework for fine-grained visual concept learning by leveraging large language model and Visual Question Answering (VQA) system. We pose these questions along with the query image to a VQA system and aggregate the answers to determine the presence or absence of an object in the test images. Our experiments demonstrate comparable performance with existing zero-shot visual classification methods and few-shot concept learning approaches.
  arXiv  Detail & Related papers  (2024-10-17T15:16:10Z)
• SADL: An Effective In-Context Learning Method for Compositional Visual QA [22.0603596548686]
  Large vision-language models (LVLMs) offer a novel capability for performing in-context learning (ICL) in Visual QA. This paper introduces SADL, a new visual-linguistic prompting framework for the task.
  arXiv  Detail & Related papers  (2024-07-02T06:41:39Z)
• BRAINTEASER: Lateral Thinking Puzzles for Large Language Models [15.95314613982879]
  BRAINTEASER is a multiple-choice Question Answering task designed to test the model's ability to exhibit lateral thinking. Our experiments with state-of-the-art instruction- and commonsense language models reveal a significant gap between human and model performance. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.
  arXiv  Detail & Related papers  (2023-10-08T07:46:01Z)
• LOIS: Looking Out of Instance Semantics for Visual Question Answering [17.076621453814926]
  We propose a model framework without bounding boxes to understand the causal nexus of object semantics in images. We implement a mutual relation attention module to model sophisticated and deeper visual semantic relations between instance objects and background information. Our proposed attention model can further analyze salient image regions by focusing on important word-related questions.
  arXiv  Detail & Related papers  (2023-07-26T12:13:00Z)
• Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering [58.64831511644917]
  We introduce an interpretable by design model that factors model decisions into intermediate human-legible explanations. We show that our inherently interpretable system can improve 4.64% over a comparable black-box system in reasoning-focused questions.
  arXiv  Detail & Related papers  (2023-05-24T08:33:15Z)
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective [93.56647950778357]
  Blind image quality assessment (BIQA) predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks.
  arXiv  Detail & Related papers  (2023-03-27T07:58:09Z)
• Cross-Modal Contrastive Learning for Robust Reasoning in VQA [76.1596796687494]
  Multi-modal reasoning in visual question answering (VQA) has witnessed rapid progress recently. Most reasoning models heavily rely on shortcuts learned from training data. We propose a simple but effective cross-modal contrastive learning strategy to get rid of the shortcut reasoning.
  arXiv  Detail & Related papers  (2022-11-21T05:32:24Z)
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [39.788346536244504]
  A-OKVQA is a crowdsourced dataset composed of about 25K questions. We demonstrate the potential of this new dataset through a detailed analysis of its contents.
  arXiv  Detail & Related papers  (2022-06-03T17:52:27Z)
• Achieving Human Parity on Visual Question Answering [67.22500027651509]
  The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image. This paper describes our recent research of AliceMind-MMU that obtains similar or even slightly better results than human beings does on VQA. This is achieved by systematically improving the VQA pipeline including: (1) pre-training with comprehensive visual and textual feature representation; (2) effective cross-modal interaction with learning to attend; and (3) A novel knowledge mining framework with specialized expert modules for the complex VQA task.
  arXiv  Detail & Related papers  (2021-11-17T04:25:11Z)
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning [132.49090098391258]
  We introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. We further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes.
  arXiv  Detail & Related papers  (2021-10-25T18:52:26Z)
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering [27.042604046441426]
  Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. In this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. We decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA.
  arXiv  Detail & Related papers  (2020-08-31T23:25:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.