A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History

• URL: http://arxiv.org/abs/2510.02362v1
• Date: Sun, 28 Sep 2025 13:03:09 GMT
• Title: A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History
• Authors: Matei-Iulian Cocu, Răzvan-Cosmin Cristia, Adrian Marius Dumitran,
• Abstract summary: The research process was carried out in three stages, to confirm the idea that the type of response expected can influence, to a certain extent, the response itself.<n>Results show that binary response stability is relatively high but far from perfect and varies by language.
• Score: 0.15293427903448023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this case study, we select a set of controversial Romanian historical questions and ask multiple Large Language Models to answer them across languages and contexts, in order to assess their biases. Besides being a study mainly performed for educational purposes, the motivation also lies in the recognition that history is often presented through altered perspectives, primarily influenced by the culture and ideals of a state, even through large language models. Since they are often trained on certain data sets that may present certain ambiguities, the lack of neutrality is subsequently instilled in users. The research process was carried out in three stages, to confirm the idea that the type of response expected can influence, to a certain extent, the response itself; after providing an affirmative answer to some given question, an LLM could shift its way of thinking after being asked the same question again, but being told to respond with a numerical value of a scale. Results show that binary response stability is relatively high but far from perfect and varies by language. Models often flip stance across languages or between formats; numeric ratings frequently diverge from the initial binary choice, and the most consistent models are not always those judged most accurate or neutral. Our research brings to light the predisposition of models to such inconsistencies, within a specific contextualization of the language for the question asked.

Related papers

• FIBER: A Multilingual Evaluation Resource for Factual Inference Bias [3.128106382761961]
  We present FIBER, a benchmark for evaluating factual knowledge in single- and multi-entity settings.<n>The dataset includes sentence completion, question-answering, and object-count prediction tasks in English, Italian, and Turkish.<n>Using FIBER, we examine whether the prompt language induces inference bias in entity selection.
  arXiv  Detail & Related papers  (2025-12-11T20:51:16Z)
• Hearing the Order: Investigating Selection Bias in Large Audio-Language Models [51.69003519291754]
  Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options.<n>In this paper, we identify and analyze this problem in LALMs.
  arXiv  Detail & Related papers  (2025-10-01T08:00:58Z)
• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG [55.258582772528506]
  We investigate whether the mixture of different document languages impacts generation and citation in unintended ways.<n>Across eight languages and six open-weight models, we find that models preferentially cite English sources when queries are in English.<n>We find that models sometimes trade-off document relevance for language preference, indicating that citation choices are not always driven by informativeness alone.
  arXiv  Detail & Related papers  (2025-09-17T12:58:18Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [45.41676783204022]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models [7.480124826347168]
  This paper investigates the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics.<n>We collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages.<n>We scrutinized the biases of these responses across languages and topics, employing two statistical hypothesis tests.
  arXiv  Detail & Related papers  (2025-05-25T12:25:44Z)
• Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs [50.07451351559251]
  We present a study across five typologically distinct languages (English, Russian, German, Hindi, and Vietnamese)<n>We examine how position bias interacts with prompt strategies and affects output entropy.
  arXiv  Detail & Related papers  (2025-05-22T02:23:00Z)
• Assessing Agentic Large Language Models in Multilingual National Bias [31.67058518564021]
  Cross-language disparities in reasoning-based recommendations remain largely unexplored.<n>This study is the first to address this gap.<n>We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages.
  arXiv  Detail & Related papers  (2025-02-25T08:07:42Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a dataset of 51.7K culturally specific questions across 23 different languages.<n>We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning [44.53966523376327]
  SeaEval is a benchmark for multilingual foundation models. We characterize how these models understand and reason with natural language. We also investigate how well they comprehend cultural practices, nuances, and values.
  arXiv  Detail & Related papers  (2023-09-09T11:42:22Z)
• Questioning the Survey Responses of Large Language Models [25.14481433176348]
  We critically examine the methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau.<n>We establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter "A"<n>Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses.
  arXiv  Detail & Related papers  (2023-06-13T17:48:27Z)
• UnQovering Stereotyping Biases via Underspecified Questions [68.81749777034409]
  We present UNQOVER, a framework to probe and quantify biases through underspecified questions. We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors. We use this metric to analyze four important classes of stereotypes: gender, nationality, ethnicity, and religion.
  arXiv  Detail & Related papers  (2020-10-06T01:49:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.