Assessing Large Language Models in Agentic Multilingual National Bias

• URL: http://arxiv.org/abs/2502.17945v1
• Date: Tue, 25 Feb 2025 08:07:42 GMT
• Title: Assessing Large Language Models in Agentic Multilingual National Bias
• Authors: Qianying Liu, Katrina Qiyao Wang, Fei Cheng, Sadao Kurohashi,
• Abstract summary: Cross-language disparities in reasoning-based recommendations remain largely unexplored.<n>This study is the first to address this gap.<n>We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages.
• Score: 31.67058518564021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, while studies on risks associated with cross biases are limited to immediate context preferences. Cross-language disparities in reasoning-based recommendations remain largely unexplored, with a lack of even descriptive analysis. This study is the first to address this gap. We test LLM's applicability and capability in providing personalized advice across three key scenarios: university applications, travel, and relocation. We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages. We quantify bias in model-generated scores and assess the impact of demographic factors and reasoning strategies (e.g., Chain-of-Thought prompting) on bias patterns. Our findings reveal that local language bias is prevalent across different tasks, with GPT-4 and Sonnet reducing bias for English-speaking countries compared to GPT-3.5 but failing to achieve robust multilingual alignment, highlighting broader implications for multilingual AI agents and applications such as education.

Related papers

• Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources [12.54580975652981]
  Large Language Models (LLMs) exhibit significant disparities in performance across languages. Continual Pretraining (CPT) has emerged as a promising approach to address this imbalance. This study systematically evaluates 36 CPT configurations involving three multilingual base models.
  arXiv  Detail & Related papers  (2025-04-05T12:10:55Z)
• The Multilingual Mind : A Survey of Multilingual Reasoning in Language Models [18.399229357408043]
  Multilingual reasoning requires language models to handle logical reasoning across languages.<n>This survey provides the first in-depth review of multilingual reasoning in Language Models.
  arXiv  Detail & Related papers  (2025-02-13T16:25:16Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? [82.02076369811402]
  Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. We present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements.
  arXiv  Detail & Related papers  (2024-03-04T10:48:13Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• On Evaluating and Mitigating Gender Biases in Multilingual Settings [5.248564173595024]
  We investigate some of the challenges with evaluating and mitigating biases in multilingual settings. We first create a benchmark for evaluating gender biases in pre-trained masked language models. We extend various debiasing methods to work beyond English and evaluate their effectiveness for SOTA massively multilingual models.
  arXiv  Detail & Related papers  (2023-07-04T06:23:04Z)
• Language-Agnostic Bias Detection in Language Models with Bias Probing [22.695872707061078]
  Pretrained language models (PLMs) are key components in NLP, but they contain strong social biases. We propose a bias probing technique called LABDet for evaluating social bias in PLMs with a robust and language-agnostic method. We find consistent patterns of nationality bias across monolingual PLMs in six languages that align with historical and political context.
  arXiv  Detail & Related papers  (2023-05-22T17:58:01Z)
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models [73.11488464916668]
  This study investigates the dynamics of the multilingual pretraining process. We probe checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.
  arXiv  Detail & Related papers  (2022-05-24T03:35:00Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment [59.995385574274785]
  We show that, contrary to previous belief, negative interference also impacts low-resource languages. We present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference.
  arXiv  Detail & Related papers  (2020-10-06T20:48:58Z)
• Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer [101.58431011820755]
  We study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications. We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations.
  arXiv  Detail & Related papers  (2020-05-02T04:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.