Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning

• URL: http://arxiv.org/abs/2605.29414v1
• Date: Thu, 28 May 2026 06:03:52 GMT
• Title: Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning
• Abstract summary: We investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese.<n>Our experiments show that simple sentence-level multilingual CSD consistently improves average multilingual performance across all four languages.
• Score: 33.701267165633105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). However, existing studies primarily focus on bilingual transfer between English and a target language, leaving multilingual settings involving three or more languages largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele. Our experiments show that simple sentence-level multilingual CSD consistently improves average multilingual performance across all four languages, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.

Related papers

• AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment [46.881574083116085]
  Multilingual large language models (LLMs) possess impressive multilingual understanding and generation capabilities.<n>LLMs' performance and cross-lingual alignment often lag for non-dominant languages.<n>We propose AlignX to bridge the multilingual performance gap, which is a two-stage representation-level framework.
  arXiv  Detail & Related papers  (2025-09-29T06:37:46Z)
• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training [58.696660064190475]
  We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities.<n>To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching.
  arXiv  Detail & Related papers  (2025-04-02T15:09:58Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods typically align vision encoders with Multimodal Large Language Models (MLLMs) via supervised fine-tuning (SFT)<n>We propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level.<n>We introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• Multilingual Instruction Tuning With Just a Pinch of Multilinguality [31.360147312195068]
  We show that many languages transfer some instruction-following capabilities to other languages from even monolingual tuning. We observe that models tuned on multilingual mixtures exhibit comparable or superior performance in multiple languages. diversifying the instruction tuning set with even just 2-4 languages significantly improves cross-lingual generalization.
  arXiv  Detail & Related papers  (2024-01-03T17:48:10Z)
• Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? [40.13166574854085]
  We investigate the minimal amount of multilinguality required to elicit cross-lingual generalisation in English-centric large language models. We find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation.
  arXiv  Detail & Related papers  (2023-12-20T00:49:52Z)
• LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation [94.33019040320507]
  Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases. We propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.
  arXiv  Detail & Related papers  (2022-10-19T12:21:39Z)
• How Do Multilingual Encoders Learn Cross-lingual Representation? [8.409283426564977]
  Cross-lingual transfer benefits languages with little to no training data by transferring from other languages. This thesis first shows such surprising cross-lingual effectiveness compared against prior art on various tasks. We also look at how to inject different cross-lingual signals into multilingual encoders, and the optimization behavior of cross-lingual transfer with these models.
  arXiv  Detail & Related papers  (2022-07-12T17:57:05Z)
• Are Multilingual Models Effective in Code-Switching? [57.78477547424949]
  We study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching.
  arXiv  Detail & Related papers  (2021-03-24T16:20:02Z)
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP [68.2650714613869]
  We propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT. Compared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages.
  arXiv  Detail & Related papers  (2020-06-11T13:15:59Z)
• Learning to Scale Multilingual Representations for Vision-Language Tasks [51.27839182889422]
  The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3-4% with less than 1/5th the training parameters compared to other word embedding methods.
  arXiv  Detail & Related papers  (2020-04-09T01:03:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.