Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors

• URL: http://arxiv.org/abs/2308.01497v3
• Date: Tue, 16 Jan 2024 19:00:56 GMT
• Title: Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors
• Authors: Nicholas Ichien, Du\v{s}an Stamenkovi\'c, Keith J. Holyoak
• Abstract summary: Large language models (LLMs) have sparked debate over whether high-level human abilities emerge in generic forms of artificial intelligence (AI) Here we assess the ability of GPT4, a state of the art large language model, to provide natural-language interpretations of novel literary metaphors. Human judges, blind to the fact that an AI model was involved, rated metaphor interpretations generated by GPT4 as superior to those provided by a group of college students.
• Score: 1.2277343096128712
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in the performance of large language models (LLMs) have sparked debate over whether, given sufficient training, high-level human abilities emerge in such generic forms of artificial intelligence (AI). Despite the exceptional performance of LLMs on a wide range of tasks involving natural language processing and reasoning, there has been sharp disagreement as to whether their abilities extend to more creative human abilities. A core example is the ability to interpret novel metaphors. Given the enormous and non curated text corpora used to train LLMs, a serious obstacle to designing tests is the requirement of finding novel yet high quality metaphors that are unlikely to have been included in the training data. Here we assessed the ability of GPT4, a state of the art large language model, to provide natural-language interpretations of novel literary metaphors drawn from Serbian poetry and translated into English. Despite exhibiting no signs of having been exposed to these metaphors previously, the AI system consistently produced detailed and incisive interpretations. Human judges, blind to the fact that an AI model was involved, rated metaphor interpretations generated by GPT4 as superior to those provided by a group of college students. In interpreting reversed metaphors, GPT4, as well as humans, exhibited signs of sensitivity to the Gricean cooperative principle. In addition, for several novel English poems GPT4 produced interpretations that were rated as excellent or good by a human literary critic. These results indicate that LLMs such as GPT4 have acquired an emergent ability to interpret complex metaphors, including those embedded in novel poems.

Related papers

• Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory [0.0]
  We show that Large Language Models (LLMs) can accurately identify and explain the presence of conceptual metaphors in natural language data. Using a novel prompting technique based on metaphor annotation guidelines, we demonstrate that LLMs are a promising tool for large-scale computational research on conceptual metaphors.
  arXiv  Detail & Related papers  (2024-10-11T17:03:13Z)
• A Perspective on Literary Metaphor in the Context of Generative AI [0.6445605125467572]
  This study explores the role of literary metaphor and its capacity to generate a range of meanings. To investigate whether the inclusion of original figurative language improves textual quality, we trained an LSTM-based language model in Afrikaans. The paper raises thought-provoking questions on aesthetic value, interpretation and evaluation.
  arXiv  Detail & Related papers  (2024-09-02T08:27:29Z)
• (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts [52.18246881218829]
  We introduce a novel multi-agent framework based on large language models (LLMs) for literary translation, implemented as a company called TransAgents. To evaluate the effectiveness of our system, we propose two innovative evaluation strategies: Monolingual Human Preference (MHP) and Bilingual LLM Preference (BLP)
  arXiv  Detail & Related papers  (2024-05-20T05:55:08Z)
• Finding Challenging Metaphors that Confuse Pretrained Language Models [21.553915781660905]
  It remains unclear what types of metaphors challenge current state-of-the-art NLP models. To identify hard metaphors, we propose an automatic pipeline that identifies metaphors that challenge a particular model. Our analysis demonstrates that our detected hard metaphors contrast significantly with VUA and reduce the accuracy of machine translation by 16%.
  arXiv  Detail & Related papers  (2024-01-29T10:00:54Z)
• Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication [68.40865217231695]
  This study examines the behavior of GPT-4V in replicating human-like use of emojis. The findings reveal a discernible discrepancy between human and GPT-4V behaviors, likely due to the subjective nature of human interpretation.
  arXiv  Detail & Related papers  (2024-01-16T08:56:52Z)
• That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? [64.38544995251642]
  We study semantic ambiguities that exist in the source (English in this work) itself. We focus on idioms that are open to both literal and figurative interpretations. We find that current MT models consistently translate English idioms literally, even when the context suggests a figurative interpretation.
  arXiv  Detail & Related papers  (2023-10-23T06:38:49Z)
• SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations [51.08119762844217]
  SenteCon is a method for introducing human interpretability in deep language representations. We show that SenteCon provides high-level interpretability at little to no cost to predictive performance on downstream tasks.
  arXiv  Detail & Related papers  (2023-05-24T05:06:28Z)
• LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models [0.0]
  Figurative language is a challenge for language models since its interpretation deviates from their conventional order and meaning. Yet, humans can easily understand and interpret metaphors as they can be derived from embodied metaphors. This study shows how larger language models perform better at interpreting metaphoric sentences when the action of the metaphorical sentence is more embodied.
  arXiv  Detail & Related papers  (2023-05-05T11:44:12Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• It's not Rocket Science : Interpreting Figurative Language in Narratives [48.84507467131819]
  We study the interpretation of two non-compositional figurative languages (idioms and similes) Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language.
  arXiv  Detail & Related papers  (2021-08-31T21:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.