CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding

• URL: http://arxiv.org/abs/2409.01389v1
• Date: Mon, 2 Sep 2024 17:39:26 GMT
• Title: CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding
• Authors: Ivana Beňová, Michal Gregor, Albert Gatt,
• Abstract summary: This study investigates the ability of various vision-language (VL) models to ground context-dependent verb phrases. We introduce the CV-Probes dataset, containing image-caption pairs with context-dependent verbs. We employ the MM-SHAP evaluation to assess the contribution of verb tokens towards model predictions.
• Score: 2.524887615873207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study investigates the ability of various vision-language (VL) models to ground context-dependent and non-context-dependent verb phrases. To do that, we introduce the CV-Probes dataset, designed explicitly for studying context understanding, containing image-caption pairs with context-dependent verbs (e.g., "beg") and non-context-dependent verbs (e.g., "sit"). We employ the MM-SHAP evaluation to assess the contribution of verb tokens towards model predictions. Our results indicate that VL models struggle to ground context-dependent verb phrases effectively. These findings highlight the challenges in training VL models to integrate context accurately, suggesting a need for improved methodologies in VL model training and evaluation.

Related papers

• VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Models [2.0718016474717196]
  integrated Vision and Language Models (VLMs) are frequently regarded as black boxes within the machine learning research community. We present an image-text aligned human visual attention dataset that maps specific associations between image regions and corresponding text segments. We then compare the internal heatmaps generated by VL models with this dataset, allowing us to analyze and better understand the model's decision-making process.
  arXiv  Detail & Related papers  (2024-10-06T20:11:53Z)
• How and where does CLIP process negation? [2.5600000778964294]
  We build on the existence task from the VALSE benchmark to test models' understanding of negation. We take inspiration from the literature on model interpretability to explain the behaviour of VL models on the understanding of negation.
  arXiv  Detail & Related papers  (2024-07-15T07:20:06Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context. We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions. We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Can Large Language Models Understand Context? [17.196362853457412]
  This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. As LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings.
  arXiv  Detail & Related papers  (2024-02-01T18:55:29Z)
• SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance [97.00445262074595]
  In SemiVL, we propose to integrate rich priors from vision-language models into semi-supervised semantic segmentation. We design a language-guided decoder to jointly reason over vision and language. We evaluate SemiVL on 4 semantic segmentation datasets, where it significantly outperforms previous semi-supervised methods.
  arXiv  Detail & Related papers  (2023-11-27T19:00:06Z)
• A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models [28.746370086515977]
  Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. We propose a framework to jointly study task performance and phrase grounding. We show how this can be addressed through brute-force training on ground phrasing annotations.
  arXiv  Detail & Related papers  (2023-09-06T03:54:57Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models [74.89629463600978]
  In vision-language domain, most large-scale pre-trained vision-language models do not possess the ability to conduct in-context learning. In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the vision domain?
  arXiv  Detail & Related papers  (2023-06-02T07:21:03Z)
• "Let's not Quote out of Context": Unified Vision-Language Pretraining for Context Assisted Image Captioning [40.01197694624958]
  We propose a new unified Vision-Language (VL) model based on the One For All (OFA) model. Our approach aims to overcome the context-independent (image and text are treated independently) nature of the existing approaches. Our system achieves state-of-the-art results with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets.
  arXiv  Detail & Related papers  (2023-06-01T17:34:25Z)
• Teaching Structured Vision&Language Concepts to Vision&Language Models [46.344585368641006]
  We introduce the collective notion of Structured Vision&Language Concepts (SVLC) SVLC includes object attributes, relations, and states which are present in the text and visible in the image. We propose a more elegant data-driven approach for enhancing VL models' understanding of SVLCs.
  arXiv  Detail & Related papers  (2022-11-21T18:54:10Z)
• On the Compositional Generalization Gap of In-Context Learning [73.09193595292233]
  We look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets.
  arXiv  Detail & Related papers  (2022-11-15T19:56:37Z)
• Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being? [0.8889304968879164]
  Vision-Language(VL) is becoming an important area of research. We propose a quantitative metric "Equivariance Score" and evaluation dataset "Human Puzzle" We aim to quantitatively measure a model's performance in understanding context.
  arXiv  Detail & Related papers  (2022-07-18T01:01:43Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• GLIPv2: Unifying Localization and Vision-Language Understanding [161.1770269829139]
  We present GLIPv2, a grounded VL understanding model, that serves both localization tasks and Vision-Language (VL) understanding tasks. GLIPv2 unifies localization pre-training and Vision-Language Pre-training with three pre-training tasks. We show that a single GLIPv2 model achieves near SoTA performance on various localization and understanding tasks.
  arXiv  Detail & Related papers  (2022-06-12T20:31:28Z)
• Does Vision-and-Language Pretraining Improve Lexical Grounding? [25.357191933430627]
  Vision-and-Language models are trained jointly on text and image or video data. It is not yet known how the internal linguistic representations themselves compare to their text-only counterparts.
  arXiv  Detail & Related papers  (2021-09-21T15:12:39Z)
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer [76.3906723777229]
  We present VidLanKD, a video-language knowledge distillation method for improving language understanding. We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset. In our experiments, VidLanKD achieves consistent improvements over text-only language models and vokenization models.
  arXiv  Detail & Related papers  (2021-07-06T15:41:32Z)
• Introducing Syntactic Structures into Target Opinion Word Extraction with Deep Learning [89.64620296557177]
  We propose to incorporate the syntactic structures of the sentences into the deep learning models for targeted opinion word extraction. We also introduce a novel regularization technique to improve the performance of the deep learning models. The proposed model is extensively analyzed and achieves the state-of-the-art performance on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-26T07:13:17Z)
• VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning [128.6138588412508]
  This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. Our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects.
  arXiv  Detail & Related papers  (2020-09-28T23:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.