Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer

• URL: http://arxiv.org/abs/2606.21954v1
• Date: Sat, 20 Jun 2026 09:03:13 GMT
• Title: Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer
• Abstract summary: Cross-lingual transfer is a model's ability to generalize capabilities from well-represented source languages to under-represented target languages.<n>Existing measures of a model's transfer strength conflate improvements in transfer with general improvements to accuracy in the source language.<n>We advocate for an alternate metric that reliably captures transfer strength called Hardness Adjusted Transfer (HAT) Score.
• Score: 38.60525517589573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-lingual transfer is a model's ability to generalize capabilities from well-represented source languages to under-represented target languages. Existing measures of a model's transfer strength conflate improvements in transfer with general improvements to accuracy in the source language. We advocate for an alternate metric that reliably captures transfer strength called Hardness Adjusted Transfer (HAT) Score, and use it to derive multiple insights on factors influencing transfer strength. Our analysis across twenty diverse language models and three popular mainstream multilingual benchmarks argues that 1) transfer in small models is not broken, 2) we are making slower than expected progress in cross-lingual transfer with model size, and 3) we have made clear progress over time.

Related papers

• Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation [49.2073409243885]
  Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency.<n>We conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages.<n>We identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages.
  arXiv  Detail & Related papers  (2026-01-01T08:53:49Z)
• LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs [67.09110757873142]
  We present LiveCLKTBench, an automated generation pipeline designed to isolate and measure cross-lingual knowledge transfer.<n>Our pipeline identifies self-contained, time-sensitive knowledge entities from real-world domains.<n>The documents of these valid entities are then used to generate factual questions, which are translated into multiple languages.
  arXiv  Detail & Related papers  (2025-11-03T17:06:49Z)
• Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks [3.274367403737527]
  Cross-lingual transfer is central to modern NLP, enabling models to perform tasks in languages different from those they were trained on.<n>A common assumption is that training on more languages improves zero-shot transfer.<n>We test this on sense-aware tasks-polysemy and lexical semantic change-and find that multilinguality is not necessary for effective transfer.
  arXiv  Detail & Related papers  (2025-05-30T17:36:20Z)
• Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability [31.025371443719404]
  Self-Translate-Train is a method that lets large language models translate training data into the target language and fine-tunes the model on its own generated data. By demonstrating that Self-Translate-Train outperforms zero-shot transfer, we encourage further exploration of better methods to elicit cross-lingual capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-06-29T14:40:23Z)
• Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer [92.80671770992572]
  Cross-lingual transfer is a central task in multilingual NLP. Earlier efforts on this task use parallel corpora, bilingual dictionaries, or other annotated alignment data. We propose a simple yet effective method, SALT, to improve the zero-shot cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-09-19T19:30:56Z)
• Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition [0.10641561702689348]
  We investigate the properties of cross-lingual transfer learning between ten low-resourced languages. We find that models that perform well on a single language often do so at the expense of generalising to others. The amount of data overlap between the source and target datasets is a better predictor of transfer performance than either the geographical or genetic distance between the languages.
  arXiv  Detail & Related papers  (2023-09-11T08:56:47Z)
• DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer [15.062937537799005]
  languages as domains for improving zero-shot transfer. We show that our approach, DiTTO, significantly outperforms the standard zero-shot fine-tuning method. Our model enables better cross-lingual transfer than standard fine-tuning methods, even in the few-shot setting.
  arXiv  Detail & Related papers  (2023-03-04T08:42:50Z)
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers [62.637055980148816]
  Massively multilingual transformers pretrained with language modeling objectives have become a de facto default transfer paradigm for NLP. We show that cross-lingual transfer via massively multilingual transformers is substantially less effective in resource-lean scenarios and for distant languages.
  arXiv  Detail & Related papers  (2020-05-01T22:04:58Z)
• Translation Artifacts in Cross-lingual Transfer Learning [51.66536640084888]
  We show that machine translation can introduce subtle artifacts that have a notable impact in existing cross-lingual models. In natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them. We also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively.
  arXiv  Detail & Related papers  (2020-04-09T17:54:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.