Finding Challenging Metaphors that Confuse Pretrained Language Models

• URL: http://arxiv.org/abs/2401.16012v1
• Date: Mon, 29 Jan 2024 10:00:54 GMT
• Title: Finding Challenging Metaphors that Confuse Pretrained Language Models
• Authors: Yucheng Li, Frank Guerin, Chenghua Lin
• Abstract summary: It remains unclear what types of metaphors challenge current state-of-the-art NLP models. To identify hard metaphors, we propose an automatic pipeline that identifies metaphors that challenge a particular model. Our analysis demonstrates that our detected hard metaphors contrast significantly with VUA and reduce the accuracy of machine translation by 16%.
• Score: 21.553915781660905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Metaphors are considered to pose challenges for a wide spectrum of NLP tasks. This gives rise to the area of computational metaphor processing. However, it remains unclear what types of metaphors challenge current state-of-the-art models. In this paper, we test various NLP models on the VUA metaphor dataset and quantify to what extent metaphors affect models' performance on various downstream tasks. Analysis reveals that VUA includes a large number of metaphors that pose little difficulty to downstream tasks. We would like to shift the attention of researchers away from these metaphors to instead focus on challenging metaphors. To identify hard metaphors, we propose an automatic pipeline that identifies metaphors that challenge a particular model. Our analysis demonstrates that our detected hard metaphors contrast significantly with VUA and reduce the accuracy of machine translation by 16\%, QA performance by 4\%, NLI by 7\%, and metaphor identification recall by over 14\% for various popular NLP systems.

Related papers

• Unveiling the Invisible: Captioning Videos with Metaphors [43.53477124719281]
  We introduce a new Vision-Language (VL) task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a dataset with 705 videos and 2115 human-written captions. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task.
  arXiv  Detail & Related papers  (2024-06-07T12:32:44Z)
• Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation [6.0158981171030685]
  We present a novel parallel dataset for the tasks of metaphor detection and interpretation that contains metaphor annotations in both Spanish and English. We investigate language models' metaphor identification and understanding abilities through a series of monolingual and cross-lingual experiments.
  arXiv  Detail & Related papers  (2024-04-10T14:44:48Z)
• Metaphor Understanding Challenge Dataset for LLMs [12.444344984005236]
  We release the Metaphor Understanding Challenge dataset (MUNCH) MUNCH is designed to evaluate the metaphor understanding capabilities of large language models (LLMs) The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases.
  arXiv  Detail & Related papers  (2024-03-18T14:08:59Z)
• Metaphorical Polysemy Detection: Conventional Metaphor meets Word Sense Disambiguation [9.860944032009847]
  Linguists distinguish between novel and conventional metaphor, a distinction which the metaphor detection task in NLP does not take into account. In this paper, we investigate the limitations of treating conventional metaphors in this way. We develop the first MPD model, which learns to identify conventional metaphors in the English WordNet.
  arXiv  Detail & Related papers  (2022-12-16T10:39:22Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• Metaphor Generation with Conceptual Mappings [58.61307123799594]
  We aim to generate a metaphoric sentence given a literal expression by replacing relevant verbs. We propose to control the generation process by encoding conceptual mappings between cognitive domains. We show that the unsupervised CM-Lex model is competitive with recent deep learning metaphor generation systems.
  arXiv  Detail & Related papers  (2021-06-02T15:27:05Z)
• MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories [5.625405679356158]
  We propose a novel metaphor detection model, namely metaphor-aware late interaction over BERT (MelBERT) Our model not only leverages contextualized word representation but also benefits from linguistic metaphor identification theories to distinguish between the contextual and literal meaning of words.
  arXiv  Detail & Related papers  (2021-04-28T07:52:01Z)
• Interpreting Verbal Metaphors by Paraphrasing [12.750941606061877]
  We show that our paraphrasing method significantly outperforms the state-of-the-art baseline. We also demonstrate that our method can help a machine translation system improve its accuracy in translating English metaphors to 8 target languages.
  arXiv  Detail & Related papers  (2021-04-07T21:00:23Z)
• Metaphoric Paraphrase Generation [58.592750281138265]
  We use crowdsourcing to evaluate our results, as well as developing an automatic metric for evaluating metaphoric paraphrases. We show that while the lexical replacement baseline is capable of producing accurate paraphrases, they often lack metaphoricity. Our metaphor masking model excels in generating metaphoric sentences while performing nearly as well with regard to fluency and paraphrase quality.
  arXiv  Detail & Related papers  (2020-02-28T16:30:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.