Can LLMs interpret figurative language as humans do?: surface-level vs representational similarity

• URL: http://arxiv.org/abs/2601.09041v1
• Date: Wed, 14 Jan 2026 00:13:00 GMT
• Title: Can LLMs interpret figurative language as humans do?: surface-level vs representational similarity
• Authors: Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi,
• Abstract summary: Human participants and four instruction-tuned LLMs rated 240 dialogue-based sentences representing six linguistic traits.<n>Results indicated that humans and LLMs aligned at the surface level with humans, but diverged significantly at the representational level.<n>GPT-4 most closely approximates human representational patterns, while all models struggle with context-dependent and socio-pragmatic expressions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models generate judgments that resemble those of humans. Yet the extent to which these models align with human judgments in interpreting figurative and socially grounded language remains uncertain. To investigate this, human participants and four instruction-tuned LLMs of different sizes (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) rated 240 dialogue-based sentences representing six linguistic traits: conventionality, sarcasm, funny, emotional, idiomacy, and slang. Each of the 240 sentences was paired with 40 interpretive questions, and both humans and LLMs rated these sentences on a 10-point Likert scale. Results indicated that humans and LLMs aligned at the surface level with humans, but diverged significantly at the representational level, especially in interpreting figurative sentences involving idioms and Gen Z slang. GPT-4 most closely approximates human representational patterns, while all models struggle with context-dependent and socio-pragmatic expressions like sarcasm, slang, and idiomacy.

Related papers

• Do Large Language Models know who did what to whom? [0.0]
  Large Language Models (LLMs) are commonly criticized for not understanding language.<n>Here, we study a kind of understanding tightly linked to language: inferring who did what to whom in a sentence.
  arXiv  Detail & Related papers  (2025-04-23T17:00:45Z)
• Iconicity in Large Language Models [0.0]
  Large language models' (LLMs') access to both meaning and sound of text is only mediated.<n>This study addresses this hypothesis by having GPT-4 generate highly iconic pseudowords in artificial languages.<n>The results revealed that humans can guess the meanings of pseudowords in the generated iconic language more accurately than words in distant natural languages.
  arXiv  Detail & Related papers  (2025-01-10T01:00:05Z)
• Language Model Alignment in Multilingual Trolley Problems [138.5684081822807]
  Building on the Moral Machine experiment, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP.<n>Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions.<n>We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems.
  arXiv  Detail & Related papers  (2024-07-02T14:02:53Z)
• Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference [1.8434042562191815]
  This work investigates the role of model scaling in determining whether differences between humans and models are amenable to model size.<n>We test three Large Language Models (LLMs) on a grammaticality judgment task featuring anaphora, center embedding, comparatives, and negative polarity.<n>We find that humans are overall less accurate than ChatGPT-4 (76% vs. 80% accuracy, respectively), but that this is due to ChatGPT-4 outperforming humans only in one task condition, namely on grammatical sentences.
  arXiv  Detail & Related papers  (2024-04-23T10:09:46Z)
• Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication [68.40865217231695]
  This study examines the behavior of GPT-4V in replicating human-like use of emojis. The findings reveal a discernible discrepancy between human and GPT-4V behaviors, likely due to the subjective nature of human interpretation.
  arXiv  Detail & Related papers  (2024-01-16T08:56:52Z)
• Divergences between Language Models and Human Brains [59.100552839650774]
  We systematically explore the divergences between human and machine language processing.<n>We identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.<n>Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.
  arXiv  Detail & Related papers  (2023-11-15T19:02:40Z)
• Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors [1.2277343096128712]
  Large language models (LLMs) have sparked debate over whether high-level human abilities emerge in generic forms of artificial intelligence (AI) Here we assess the ability of GPT4, a state of the art large language model, to provide natural-language interpretations of novel literary metaphors. Human judges, blind to the fact that an AI model was involved, rated metaphor interpretations generated by GPT4 as superior to those provided by a group of college students.
  arXiv  Detail & Related papers  (2023-08-03T01:46:27Z)
• Speaking Multiple Languages Affects the Moral Bias of Language Models [70.94372902010232]
  Pre-trained multilingual language models (PMLMs) are commonly used when dealing with data from multiple languages and cross-lingual transfer. Do the models capture moral norms from English and impose them on other languages? Our experiments demonstrate that, indeed, PMLMs encode differing moral biases, but these do not necessarily correspond to cultural differences or commonalities in human opinions.
  arXiv  Detail & Related papers  (2022-11-14T20:08:54Z)
• Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models [91.3755431537592]
  Representing compositional and non-compositional phrases is critical for language understanding. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case.
  arXiv  Detail & Related papers  (2022-10-07T14:21:30Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Do Multilingual Language Models Capture Differing Moral Norms? [71.52261949766101]
  Massively multilingual sentence representations are trained on large corpora of uncurated data. This may cause the models to grasp cultural values including moral judgments from the high-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs.
  arXiv  Detail & Related papers  (2022-03-18T12:26:37Z)
• It's not Rocket Science : Interpreting Figurative Language in Narratives [48.84507467131819]
  We study the interpretation of two non-compositional figurative languages (idioms and similes) Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language.
  arXiv  Detail & Related papers  (2021-08-31T21:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.