Disentangling Language Roles in Multilingual LLM Task Execution

• URL: http://arxiv.org/abs/2605.27649v1
• Date: Tue, 26 May 2026 20:09:34 GMT
• Title: Disentangling Language Roles in Multilingual LLM Task Execution
• Abstract summary: MTM-Bench is a benchmark for language-conditioned task execution.<n>It enumerates all 27 triplets and contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity.<n>We evaluate 20 frontier and open-weight LLMs using metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success.
• Score: 17.182371695349385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.

Related papers

• Better Decomposition, Free Aggregation: A Synthesizer-Folding Framework for Multilingual Multi-Hop Question Answering [48.94279718429342]
  We introduce Syfer, a framework for multilingual multi-hop question answering.<n>Syfer defers translation rather than applying it by default.<n> Experiments across multiple languages show that Syfer attains competitive accuracy.
  arXiv  Detail & Related papers  (2026-08-13T12:25:59Z)
• Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning [61.568735270400786]
  Large language models (LLMs) perform strongly across tasks and languages.<n>How improvements in one task or language affect other tasks and languages and their combinations remains poorly understood.
  arXiv  Detail & Related papers  (2025-11-17T13:41:31Z)
• Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches [8.127643463046516]
  We examine strategies to improve the multilingual and crosslingual performance.<n>We evaluate approaches on a dataset containing posts and claims in 47 languages.<n>Most importantly, we show that crosslinguality is a setup with its own unique characteristics compared to the multilingual setup.
  arXiv  Detail & Related papers  (2025-05-28T08:47:10Z)
• Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate [36.641755706551336]
  Large language models (LLMs) provide detailed and impressive responses to queries in English.<n>But are they really consistent at responding to the same query in other languages?<n>We propose a framework to evaluate LLM's cross-lingual consistency based on a simple Translate then Evaluate strategy.
  arXiv  Detail & Related papers  (2025-05-28T06:00:21Z)
• Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs [12.334510055293535]
  Large language models (LLMs) pre-trained predominantly on English text exhibit surprising multilingual capabilities.<n>We introduce cross-lingual alignment metrics to quantify the alignment at an instance level for discriminative tasks.<n>We find that while cross-lingual alignment metrics strongly correlate with task accuracy at the language level, the sample-level alignment often fails to distinguish correct from incorrect predictions.
  arXiv  Detail & Related papers  (2025-04-13T00:01:22Z)
• Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following [51.18383180774354]
  We introduce Multi-IF, a new benchmark designed to assess Large Language Models' proficiency in following multi-turn and multilingual instructions. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities.
  arXiv  Detail & Related papers  (2024-10-21T00:59:47Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation [113.99145386490639]
  Cross-lingual NER can transfer knowledge between languages via aligned cross-lingual representations or machine translation results. We propose a Cross-lingual Entity Projection framework (CROP) to enable zero-shot cross-lingual NER. We adopt a multilingual labeled sequence translation model to project the tagged sequence back to the target language and label the target raw sentence.
  arXiv  Detail & Related papers  (2022-10-13T13:32:36Z)
• Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation [80.16548523140025]
  We extend the vanilla pretrain-finetune pipeline with extra code-switching restore task to bridge the gap between the pretrain and finetune stages. Our approach could narrow the cross-lingual sentence representation distance and improve low-frequency word translation with trivial computational cost.
  arXiv  Detail & Related papers  (2022-04-16T16:08:38Z)
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment [71.53159402053392]
  We propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios.
  arXiv  Detail & Related papers  (2020-09-30T08:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.