Comparative Study of Multilingual Idioms and Similes in Large Language Models

• URL: http://arxiv.org/abs/2410.16461v1
• Date: Mon, 21 Oct 2024 19:40:05 GMT
• Title: Comparative Study of Multilingual Idioms and Similes in Large Language Models
• Authors: Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, Yadollah Yaghoobzadeh,
• Abstract summary: This study explores the effectiveness of prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model.
• Score: 4.581124233698535
• License:
• Abstract: This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.

Related papers

• Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense [30.62699081329474]
  We introduce a novel benchmark for cross-lingual sense disambiguation, StingrayBench. We collect false friends in four language pairs, namely Indonesian-Malay, Indonesian-Tagalog, Chinese-Japanese, and English-German. In our analysis of various models, we observe they tend to be biased toward higher-resource languages.
  arXiv  Detail & Related papers  (2024-10-28T22:09:43Z)
• Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysis [19.37853222555255]
  Probing techniques for large language models (LLMs) have primarily focused on English, overlooking the vast majority of the world's languages. We conduct experiments on several open-source LLM models, analyzing probing accuracy, trends across layers, and similarities between probing vectors for multiple languages.
  arXiv  Detail & Related papers  (2024-09-22T14:14:05Z)
• The Less the Merrier? Investigating Language Representation in Multilingual Models [8.632506864465501]
  We investigate the linguistic representation of different languages in multilingual models. We observe from our experiments that community-centered models perform better at distinguishing between languages in the same family for low-resource languages.
  arXiv  Detail & Related papers  (2023-10-20T02:26:34Z)
• Exploring the Maze of Multilingual Modeling [2.0849578298972835]
  We present a comprehensive evaluation of three popular multilingual language models: mBERT, XLM-R, and GPT-3. Our findings reveal that while the amount of language-specific pretraining data plays a crucial role in model performance, we also identify other factors such as general resource availability, language family, and script type, as important features.
  arXiv  Detail & Related papers  (2023-10-09T04:48:14Z)
• Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples [89.16814518860357]
  The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language.
  arXiv  Detail & Related papers  (2023-03-30T16:34:10Z)
• Multilingual Representation Distillation with Contrastive Learning [20.715534360712425]
  We integrate contrastive learning into multilingual representation distillation and use it for quality estimation of parallel sentences. We validate our approach with multilingual similarity search and corpus filtering tasks.
  arXiv  Detail & Related papers  (2022-10-10T22:27:04Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Specializing Multilingual Language Models: An Empirical Study [50.7526245872855]
  Contextualized word representations from pretrained multilingual language models have become the de facto standard for addressing natural language tasks. For languages rarely or never seen by these models, directly using such models often results in suboptimal representation or use of data.
  arXiv  Detail & Related papers  (2021-06-16T18:13:55Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.