Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages

• URL: http://arxiv.org/abs/2604.27137v1
• Date: Wed, 29 Apr 2026 19:48:58 GMT
• Title: Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages
• Abstract summary: This paper introduces a systematic evaluation framework grounded in the Interagency Language Roundtable Skill Level Descriptions.<n>We administer a battery of 12 semantically equivalent prompt clusters across six languages: English, French, Romanian, Spanish, Italian, and German.<n>We analyze outputs through a two-layer methodology combining automated quantitative metrics with expert ILR qualitative assessment.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a systematic evaluation framework grounded in the Interagency Language Roundtable (ILR) Skill Level Descriptions and applies it to Claude (Sonnet 4.6) across six languages: English, French, Romanian, Spanish, Italian, and German. We administer a battery of 12 semantically equivalent prompt clusters spanning ILR complexity levels 1 through 3+, collect 216 responses (12 prompts, 6 languages, 3 runs), and analyze outputs through a two-layer methodology combining automated quantitative metrics with expert ILR qualitative assessment. Quantitative analysis reveals that French responses are approximately 30% longer than German responses on identical prompts, and that creative and affective clusters show the highest cross-lingual surface divergence. Qualitative analysis, conducted by a six-language professional with 12 years of ILR/OPI assessment experience, identifies five cross-lingual variation patterns: systematic differences in pragmatic disambiguation strategies, aesthetic and literary tradition divergence in creative output, language-internal technical terminology norms, cultural calibration gaps evidenced by the absence of culture-specific content in favor of culturally neutralized templates, and language-specific institutional referral behavior in emotional support responses. We argue that ILR-informed expert judgment applied to LLM outputs constitutes a novel and underreported evaluation methodology that complements purely computational benchmarks, and that cross-lingual output variation in Claude is interpretable, domain-dependent, and consequential for equitable multilingual AI deployment.

Related papers

• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis [2.8523456804049885]
  We construct paired DT and culturally-adapted datasets via 1:1 seed matching for four languages.<n>We compare Attack Success Rate (ASR) and Cultural Realism scores across four open-source safety evaluations.
  arXiv  Detail & Related papers  (2026-06-08T08:17:18Z)
• Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation [65.69228479949693]
  Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages.<n>We propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expertworks.
  arXiv  Detail & Related papers  (2026-04-04T04:56:35Z)
• Gradient-Informed Training for Low-Resource Multilingual Speech Translation [4.344398827902069]
  This work proposes a principled methodology to automatically determine layer-specific sharing patterns by mining training gradient information.<n>Our approach employs three distinct analysis strategies: distance-based language clustering, self/cross-task divergence metrics for capacity allocation, and joint factorization.
  arXiv  Detail & Related papers  (2026-03-26T19:00:00Z)
• Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish [12.286855282078305]
  GPT-4o, GPT-4, Claude3.5Sonnet, LLaMA3.1, MistralLarge2, LLaMA-2Chat13B, and Mistral7BInstruct are evaluated.<n>Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue.
  arXiv  Detail & Related papers  (2025-11-05T22:09:53Z)
• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models [64.54005959758733]
  We introduce code-switching in-context learning (CSICL) as a principled and robust approach for overcoming the translation barrier during inference.<n>We conduct extensive experiments across 4 LLMs, 6 datasets, and 10 languages, spanning both knowledge-intensive and reasoning-oriented domains.<n>Our results demonstrate CSICL consistently outperforms X-ICL baselines, achieving gains of 3.1%p and 1.9%p in both target and unseen languages.
  arXiv  Detail & Related papers  (2025-10-07T08:35:42Z)
• XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering [48.913480244527925]
  Large Language Models (LLMs) have shown significant progress in Open-domain question answering (ODQA)<n>Most evaluations focus on English and assume locale-invariant answers across languages.<n>We introduce XLQA, a novel benchmark explicitly designed for locale-sensitive multilingual ODQA.
  arXiv  Detail & Related papers  (2025-08-22T07:00:13Z)
• Disentangling Language and Culture for Evaluating Multilingual Large Language Models [48.06219053598005]
  This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs.<n>By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions cross-lingually.
  arXiv  Detail & Related papers  (2025-05-30T14:25:45Z)
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation [86.7047714187813]
  MMLU-ProX is a benchmark covering 29 languages, built on an English benchmark.<n>Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons.<n>To meet efficient evaluation needs, we provide a lite version containing 658 questions per language.
  arXiv  Detail & Related papers  (2025-03-13T15:59:20Z)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following [59.549015333755186]
  Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications.<n>Existing evaluations lack fine-grained constraint analysis across diverse linguistic contexts.<n>We introduce XIFBench, a comprehensive benchmark for evaluating multilingual instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-10T17:07:52Z)
• Beyond No: Quantifying AI Over-Refusal and Emotional Attachment Boundaries [0.0]
  We present an open-source benchmark and evaluation framework for assessing emotional boundary handling in Large Language Models (LLMs)<n>We evaluated three leading LLMs on their ability to maintain appropriate emotional boundaries through pattern-matched response analysis.<n>Our framework quantifies responses across seven key patterns: direct refusal, apology, explanation, deflection, acknowledgment, boundary setting, and emotional awareness.
  arXiv  Detail & Related papers  (2025-02-20T19:09:40Z)
• MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models [0.5822010906632046]
  This study introduces MultiPragEval, the first pragmatic evaluation of Large Language Models (LLMs) Comprising 1200 question units categorized according to Grice's Cooperative Principle, MultiPragEval enables an in-depth assessment of LLMs' contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field.
  arXiv  Detail & Related papers  (2024-06-11T21:46:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.