Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs

• URL: http://arxiv.org/abs/2505.16134v2
• Date: Fri, 26 Sep 2025 15:21:49 GMT
• Title: Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs
• Authors: Mikhail Menschikov, Alexander Kharitonov, Maiia Kotyga, Vadim Porvatov, Anna Zhukovskaya, David Kagramanyan, Egor Shvetsov, Evgeny Burnaev,
• Abstract summary: We present a study across five typologically distinct languages (English, Russian, German, Hindi, and Vietnamese)<n>We examine how position bias interacts with prompt strategies and affects output entropy.
• Score: 50.07451351559251
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) exhibit position bias - a systematic tendency to neglect information at specific context positions. However, the patterns of position bias behavior, depending on the language or model, remain unexplored. We present a multilingual study across five typologically distinct languages (English, Russian, German, Hindi, and Vietnamese) and five model architectures, examining how position bias interacts with prompt strategies and affects output entropy. Our key findings are: (1) Position bias is primarily model-driven, yet exhibits language-specific variations. For instance, Qwen2.5-7B-Instruct and DeepSeek 7B Chat consistently favors late positions, challenging established assumptions of a universal early-token bias in LLMs. (2) Explicitly instructing the model that "the context is relevant to the query" unexpectedly reduces accuracy across languages, undermining common prompt-engineering practices. (3) While the largest accuracy drop occurs when relevant information is placed in the middle of the context, this is not explicitly reflected by a corresponding peak in output entropy.

Related papers

• Cross-Language Bias Examination in Large Language Models [37.21579885190632]
  This study introduces an innovative multilingual bias evaluation framework for assessing bias in Large Language Models.<n>By translating the prompts and word list into five target languages, we compare different types of bias across languages.<n>For example, Arabic and Spanish consistently show higher levels of stereotype bias, while Chinese and English exhibit lower levels of bias.
  arXiv  Detail & Related papers  (2025-12-17T23:22:03Z)
• FIBER: A Multilingual Evaluation Resource for Factual Inference Bias [3.128106382761961]
  We present FIBER, a benchmark for evaluating factual knowledge in single- and multi-entity settings.<n>The dataset includes sentence completion, question-answering, and object-count prediction tasks in English, Italian, and Turkish.<n>Using FIBER, we examine whether the prompt language induces inference bias in entity selection.
  arXiv  Detail & Related papers  (2025-12-11T20:51:16Z)
• Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models [7.480124826347168]
  This paper investigates the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics.<n>We collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages.<n>We scrutinized the biases of these responses across languages and topics, employing two statistical hypothesis tests.
  arXiv  Detail & Related papers  (2025-05-25T12:25:44Z)
• Assessing Large Language Models in Agentic Multilingual National Bias [31.67058518564021]
  Cross-language disparities in reasoning-based recommendations remain largely unexplored.<n>This study is the first to address this gap.<n>We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages.
  arXiv  Detail & Related papers  (2025-02-25T08:07:42Z)
• Covert Bias: The Severity of Social Views' Unalignment in Language Models Towards Implicit and Explicit Opinion [0.40964539027092917]
  We evaluate the severity of bias toward a view by using a biased model in edge cases of excessive bias scenarios. Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances. The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness.
  arXiv  Detail & Related papers  (2024-08-15T15:23:00Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Mitigate Position Bias in Large Language Models via Scaling a Single Dimension [47.792435921037274]
  This paper first explores the micro-level manifestations of position bias, concluding that attention weights are a micro-level expression of position bias.<n>It further identifies that, in addition to position embeddings, causal attention mask also contributes to position bias by creating position-specific hidden states.<n>Based on these insights, we propose a method to mitigate position bias by scaling this positional hidden states.
  arXiv  Detail & Related papers  (2024-06-04T17:55:38Z)
• SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning [44.53966523376327]
  SeaEval is a benchmark for multilingual foundation models. We characterize how these models understand and reason with natural language. We also investigate how well they comprehend cultural practices, nuances, and values.
  arXiv  Detail & Related papers  (2023-09-09T11:42:22Z)
• Language-Agnostic Bias Detection in Language Models with Bias Probing [22.695872707061078]
  Pretrained language models (PLMs) are key components in NLP, but they contain strong social biases. We propose a bias probing technique called LABDet for evaluating social bias in PLMs with a robust and language-agnostic method. We find consistent patterns of nationality bias across monolingual PLMs in six languages that align with historical and political context.
  arXiv  Detail & Related papers  (2023-05-22T17:58:01Z)
• A Survey on Zero Pronoun Translation [69.09774294082965]
  Zero pronouns (ZPs) are frequently omitted in pro-drop languages, but should be recalled in non-pro-drop languages. This survey paper highlights the major works that have been undertaken in zero pronoun translation (ZPT) after the neural revolution. We uncover a number of insightful findings such as: 1) ZPT is in line with the development trend of large language model; 2) data limitation causes learning bias in languages and domains; 3) performance improvements are often reported on single benchmarks, but advanced methods are still far from real-world use.
  arXiv  Detail & Related papers  (2023-05-17T13:19:01Z)
• The Curious Case of Absolute Position Embeddings [65.13827063579728]
  Transformer language models encode the notion of word order using positional information. In natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. We observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information.
  arXiv  Detail & Related papers  (2022-10-23T00:00:04Z)
• Revisiting the Uniform Information Density Hypothesis [44.277066511088634]
  We investigate the uniform information density (UID) hypothesis using reading time and acceptability data. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability.
  arXiv  Detail & Related papers  (2021-09-23T20:41:47Z)
• The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models [11.148662334602639]
  We analyze the position embeddings of existing language models and find strong evidence of translation invariance. We propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion.
  arXiv  Detail & Related papers  (2021-06-03T15:56:26Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model [66.84264870118723]
  We present the first purely corpus-driven model of multi-lingual adjective ordering in the form of a latent-variable model. We provide strong converging evidence for the existence of universal, cross-linguistic, hierarchical adjective ordering tendencies.
  arXiv  Detail & Related papers  (2020-10-09T18:27:55Z)
• UnQovering Stereotyping Biases via Underspecified Questions [68.81749777034409]
  We present UNQOVER, a framework to probe and quantify biases through underspecified questions. We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors. We use this metric to analyze four important classes of stereotypes: gender, nationality, ethnicity, and religion.
  arXiv  Detail & Related papers  (2020-10-06T01:49:52Z)
• Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer [101.58431011820755]
  We study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications. We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations.
  arXiv  Detail & Related papers  (2020-05-02T04:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.