Extending Beacon to Hindi: Cultural Adaptation Drives Cross-Lingual Sycophancy

• URL: http://arxiv.org/abs/2602.00046v1
• Date: Mon, 19 Jan 2026 12:03:52 GMT
• Title: Extending Beacon to Hindi: Cultural Adaptation Drives Cross-Lingual Sycophancy
• Abstract summary: Sycophancy is the tendency of language models to prioritize agreement with user preferences over principled reasoning.<n>We extend the Beacon single-turn forced-choice sycophancy diagnostic to Hindi through a controlled three-condition design.<n>Across all models, sycophancy rates are consistently higher for culturally adapted Hindi prompts than for English.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sycophancy, the tendency of language models to prioritize agreement with user preferences over principled reasoning, has been identified as a persistent alignment failure in English-language evaluations. However, it remains unclear whether such diagnostics generalize across languages and cultural contexts. We extend the Beacon single-turn forced-choice sycophancy diagnostic to Hindi through a controlled three-condition design: English original, Hindi literal translation, and Hindi culturally adapted prompts. We evaluate four open-weight instruction-tuned models on 50 prompts per condition, enabling separation of language encoding effects from cultural adaptation effects. Across all models, sycophancy rates are consistently higher for culturally adapted Hindi prompts than for English, with absolute differences ranging from 12.0 to 16.0 percentage points. A decomposition on Qwen 2.5-Coder-7B shows that cultural adaptation (delta = 14.0%, 95% CI: [4.0%, 26.0%]) accounts for the majority of this gap, while language encoding contributes minimally (delta = 2.0%, 95% CI: [0.0%, 6.0%]). Category-level analysis reveals that advice prompts exhibit the largest cross-lingual differences (20-25 percentage points), achieving statistical significance in two of four models. These findings indicate that alignment behaviors measured in English may not transfer uniformly across languages and that culturally grounded prompt framing plays a substantial role. We release all datasets and evaluation code to support replication and extension.

Related papers

• On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation [79.18001266925245]
  LingT2I is a benchmark covering 10 widely used languages with 33K prompts.<n>We conduct a comprehensive cross-lingual analysis, uncovering linguistic inequality and language-dependent trade-offs.
  arXiv  Detail & Related papers  (2026-08-11T14:49:23Z)
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis [2.8523456804049885]
  We construct paired DT and culturally-adapted datasets via 1:1 seed matching for four languages.<n>We compare Attack Success Rate (ASR) and Cultural Realism scores across four open-source safety evaluations.
  arXiv  Detail & Related papers  (2026-06-08T08:17:18Z)
• The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models [51.61416200800499]
  GaoYao is a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas.<n>First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers.<n>Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages.<n>Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs.
  arXiv  Detail & Related papers  (2026-04-22T06:19:46Z)
• Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation [9.245983423391364]
  We evaluate 60 culturally grounded conversational scenarios across five languages.<n>We score responses on 12 pragmatic features covering deference to authority, individual-versus-group framing, and uncertainty management.
  arXiv  Detail & Related papers  (2026-04-20T01:58:57Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• Simultaneous Speech-to-Speech Translation Without Aligned Data [52.467808474293605]
  Simultaneous speech translation requires translating source speech into a target language in real-time.<n>We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely.<n>Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks.
  arXiv  Detail & Related papers  (2026-02-11T17:41:01Z)
• "Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs [0.0]
  We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation.<n>We evaluate 7 multilingual large language models (LLMs) across 15 target languages with 5 native-speaker raters per language.<n>Across full-text evaluations, mean overall quality is modest (1.68/3): GPT-5 (2.10/3), Claude Sonnet 3.7 (1.97/3), and Mistral Medium 3.1 (1.84/3) form the strongest tier with fewer catastrophic failures.
  arXiv  Detail & Related papers  (2026-02-04T16:35:48Z)
• The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining [29.376308590290297]
  We compare the standard web corpus with a monolingual-only version that removes all multilingual documents.<n>We categorize bilingual data into parallel (14%), code-switching (72%), and miscellaneous documents (14%) based on semantic relevance.<n>Our experiments reveal that parallel data almost fully restores translation performance, whereas code-switching contributes minimally.
  arXiv  Detail & Related papers  (2026-01-01T14:52:06Z)
• Do Large Language Models Truly Understand Cross-cultural Differences? [53.481048019144644]
  We develop a scenario-based benchmark to evaluate large language models' cross-cultural understanding and reasoning.<n>Grounded in cultural theory, we categorize cross-cultural capabilities into nine dimensions.<n>The dataset supports continuous expansion, and experiments confirm its transferability to other languages.
  arXiv  Detail & Related papers  (2025-12-08T01:21:58Z)
• CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models [0.42970700836450487]
  We introduce CRaFT, an explanation-based multilingual evaluation framework designed to assess how large language models (LLMs) reason across cultural contexts.<n>We apply the framework to 50 culturally grounded questions from the World Values Survey, translated into Arabic, Bengali, and Spanish, and evaluate three models (GPT, DeepSeek, and FANAR) across over 2,100 answer-explanation pairs.<n>Results reveal significant cross-lingual variation in reasoning: Arabic reduces fluency, Bengali enhances it, and Spanish remains largely stable.
  arXiv  Detail & Related papers  (2025-10-15T18:49:10Z)
• PARAM-1 BharatGen 2.9B Model [14.552007884700618]
  PARAM-1 is a 2.9B parameter decoder-only, text-only language model trained from scratch with an explicit architectural and linguistic focus on Indian diversity.<n>It is guided by three core principles: equitable representation of Indic languages through a 25% corpus allocation; tokenization fairness via a SentencePiece tokenizer adapted to Indian morphological structures; and culturally aligned evaluation benchmarks across IndicQA, code-mixed reasoning, and socio-linguistic robustness tasks.
  arXiv  Detail & Related papers  (2025-07-16T06:14:33Z)
• JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community [9.492476871323763]
  This paper introduces JiraiBench, the first bilingual benchmark for evaluating large language models' effectiveness in detecting self-destructive content.<n>We focus on the transnational "Jirai" (landmine) online subculture that encompasses multiple forms of self-destructive behaviors including drug overdose, eating disorders, and self-harm.<n>Our dataset comprises 10,419 Chinese posts and 5,000 Japanese posts with multidimensional annotation along three behavioral categories.
  arXiv  Detail & Related papers  (2025-03-27T16:48:58Z)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [68.21939124278065]
  Culturally-diverse multilingual Visual Question Answering benchmark designed to cover a rich set of languages and cultures. CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-06-10T01:59:00Z)
• Self-Consistency Improves Chain of Thought Reasoning in Language Models [53.45015291520658]
  We explore a simple ensemble strategy, self-consistency, that significantly improves the reasoning accuracy of large language models. For arithmetic and commonsense reasoning benchmarks we find that self-consistency yields significant accuracy improvements.
  arXiv  Detail & Related papers  (2022-03-21T17:48:52Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
  Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
  arXiv  Detail & Related papers  (2020-05-01T12:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.