CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding

• URL: http://arxiv.org/abs/2409.01389v1
• Date: Mon, 2 Sep 2024 17:39:26 GMT
• Title: CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding
• Authors: Ivana Beňová, Michal Gregor, Albert Gatt,
• Abstract summary: This study investigates the ability of various vision-language (VL) models to ground context-dependent verb phrases. We introduce the CV-Probes dataset, containing image-caption pairs with context-dependent verbs. We employ the MM-SHAP evaluation to assess the contribution of verb tokens towards model predictions.
• Score: 2.524887615873207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study investigates the ability of various vision-language (VL) models to ground context-dependent and non-context-dependent verb phrases. To do that, we introduce the CV-Probes dataset, designed explicitly for studying context understanding, containing image-caption pairs with context-dependent verbs (e.g., "beg") and non-context-dependent verbs (e.g., "sit"). We employ the MM-SHAP evaluation to assess the contribution of verb tokens towards model predictions. Our results indicate that VL models struggle to ground context-dependent verb phrases effectively. These findings highlight the challenges in training VL models to integrate context accurately, suggesting a need for improved methodologies in VL model training and evaluation.

Related papers

• VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Models [2.0718016474717196]
  integrated Vision and Language Models (VLMs) are frequently regarded as black boxes within the machine learning research community. We present an image-text aligned human visual attention dataset that maps specific associations between image regions and corresponding text segments. We then compare the internal heatmaps generated by VL models with this dataset, allowing us to analyze and better understand the model's decision-making process.
  arXiv  Detail & Related papers  (2024-10-06T20:11:53Z)
• How and where does CLIP process negation? [2.5600000778964294]
  We build on the existence task from the VALSE benchmark to test models' understanding of negation. We take inspiration from the literature on model interpretability to explain the behaviour of VL models on the understanding of negation.
  arXiv  Detail & Related papers  (2024-07-15T07:20:06Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context. We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions. We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Can Large Language Models Understand Context? [17.196362853457412]
  This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. As LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings.
  arXiv  Detail & Related papers  (2024-02-01T18:55:29Z)
• A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models [28.746370086515977]
  Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. We propose a framework to jointly study task performance and phrase grounding. We show how this can be addressed through brute-force training on ground phrasing annotations.
  arXiv  Detail & Related papers  (2023-09-06T03:54:57Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models [74.89629463600978]
  In vision-language domain, most large-scale pre-trained vision-language models do not possess the ability to conduct in-context learning. In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the vision domain?
  arXiv  Detail & Related papers  (2023-06-02T07:21:03Z)
• On the Compositional Generalization Gap of In-Context Learning [73.09193595292233]
  We look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets.
  arXiv  Detail & Related papers  (2022-11-15T19:56:37Z)
• Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being? [0.8889304968879164]
  Vision-Language(VL) is becoming an important area of research. We propose a quantitative metric "Equivariance Score" and evaluation dataset "Human Puzzle" We aim to quantitatively measure a model's performance in understanding context.
  arXiv  Detail & Related papers  (2022-07-18T01:01:43Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• Introducing Syntactic Structures into Target Opinion Word Extraction with Deep Learning [89.64620296557177]
  We propose to incorporate the syntactic structures of the sentences into the deep learning models for targeted opinion word extraction. We also introduce a novel regularization technique to improve the performance of the deep learning models. The proposed model is extensively analyzed and achieves the state-of-the-art performance on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-26T07:13:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.