Beyond Surface Cues: Disentangling Sociocultural Signals in Multilingual LLMs

• URL: http://arxiv.org/abs/2608.23026v1
• Date: Mon, 24 Aug 2026 09:32:16 GMT
• Title: Beyond Surface Cues: Disentangling Sociocultural Signals in Multilingual LLMs
• Abstract summary: The study analyzes 89,253 outputs from 12 Multilingual LLMs in English, French, and Chinese.<n>Removing direct identity cues reduces identity-label prediction in English and Chinese, but has a much smaller effect in French.<n>The ability to identify the source language drops substantially after translation and again after masking names.
• Score: 28.912535376913436
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual LLM outputs can vary across sociocultural contexts. However, evidence of cultural grounding can be misleading: identity labels may be inferred from explicit or indirect textual cues, while names and wording can reveal the source language. Treating all these signals as evidence of cultural grounding may obscure potential biases. We present a human-validated, multi-agent audit that separates three questions: whether outputs reproduce social biases, whether identity groups are represented differently, and whether outputs reflect cross-cultural patterns. The study analyzes 89,253 outputs from 12 LLMs in English, French, and Chinese, spanning 18 occupations and three task conditions. We find that bias representation varies systematically across languages and tasks. Removing direct identity cues sharply reduces identity-label prediction in English and Chinese, but has a much smaller effect in French. Across all language-genre settings, the cultural context associated with the source language receives the highest average relevance score, with moderate agreement between automated and human ratings. However, the ability to identify the source language drops substantially after translation and again after masking names. Without these controls, multilingual audits may mistake surface cues for cultural understanding, leading to misleading conclusions about cross-cultural variation and bias. Our audit offers a practical framework for separating such shortcuts from more meaningful cross-cultural patterns.

Related papers

• MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark [31.67992649991894]
  A model that can speak a user's language must also understand the culture encoded by that language.<n>We introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers.<n>We find substantial cultural degradation and a pronounced Locality Effect.
  arXiv  Detail & Related papers  (2026-07-01T10:12:03Z)
• LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing [0.21485350418225244]
  Large Language Models (LLMs) are rapidly being adopted by users across the globe, who interact with them in a diverse range of languages.<n>We examine how prompt language and cultural framing influence model responses and their alignment with human values in different countries.
  arXiv  Detail & Related papers  (2025-11-06T02:09:29Z)
• Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge [68.6805229085352]
  Most multilingual question-answering benchmarks do not factor in regional diversity in the information they capture.<n>XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages.<n>We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics.
  arXiv  Detail & Related papers  (2025-11-01T18:41:34Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models [2.48490797934472]
  Artificial Intelligence (AI) is increasingly used in hiring, with large language models (LLMs) having the potential to influence or even make hiring decisions.<n>This raises pressing concerns about bias, fairness, and trust, particularly across diverse cultural contexts.<n>We conduct a systematic analysis of how LLMs assess job interviews across cultural and identity dimensions.
  arXiv  Detail & Related papers  (2025-08-21T07:45:00Z)
• See It from My Perspective: How Language Affects Cultural Bias in Image Understanding [60.70852566256668]
  Vision-language models (VLMs) can respond to queries about images in many languages.<n>We characterize the Western bias of VLMs in image understanding and investigate the role that language plays in this disparity.
  arXiv  Detail & Related papers  (2024-06-17T15:49:51Z)
• The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning [23.418656688405605]
  We study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.
  arXiv  Detail & Related papers  (2024-05-21T12:55:15Z)
• Multi-lingual and Multi-cultural Figurative Language Understanding [69.47641938200817]
  Figurative language permeates human communication, but is relatively understudied in NLP. We create a dataset for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data.
  arXiv  Detail & Related papers  (2023-05-25T15:30:31Z)
• Comparing Biases and the Impact of Multilingual Training across Multiple Languages [70.84047257764405]
  We present a bias analysis across Italian, Chinese, English, Hebrew, and Spanish on the downstream sentiment analysis task. We adapt existing sentiment bias templates in English to Italian, Chinese, Hebrew, and Spanish for four attributes: race, religion, nationality, and gender. Our results reveal similarities in bias expression such as favoritism of groups that are dominant in each language's culture.
  arXiv  Detail & Related papers  (2023-05-18T18:15:07Z)
• Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure [54.01613740115601]
  We study three language properties: constituent order, composition and word co-occurrence. Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
  arXiv  Detail & Related papers  (2022-03-16T07:09:35Z)
• Deception detection in text and its relation to the cultural dimension of individualism/collectivism [6.17866386107486]
  We investigate if differences in the usage of specific linguistic features of deception across cultures can be confirmed and attributed to norms in respect to the individualism/collectivism divide. We create culture/language-aware classifiers by experimenting with a wide range of n-gram features based on phonology, morphology and syntax. We conducted our experiments over 11 datasets from 5 languages i.e., English, Dutch, Russian, Spanish and Romanian, from six countries (US, Belgium, India, Russia, Mexico and Romania)
  arXiv  Detail & Related papers  (2021-05-26T13:09:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.