XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation

• URL: http://arxiv.org/abs/2412.12686v1
• Date: Tue, 17 Dec 2024 09:05:30 GMT
• Title: XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation
• Authors: Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Libo Qin, Yichong Huang, Lei Huang, Weitao Ma, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin,
• Abstract summary: Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability. We propose a probing method named XTransplant that explores cross-lingual latent interactions via cross-lingual feed-forward transplantation. We empirically prove that both the multilingual capabilities and cultural adaptability of LLMs hold the potential to be significantly improved by XTransplant.
• Score: 49.69780199602105
• License:
• Abstract: Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability, largely due to their English-centric pretraining data. To address this imbalance, we propose a probing method named XTransplant that explores cross-lingual latent interactions via cross-lingual feed-forward transplantation during inference stage, with the hope of enabling the model to leverage the strengths of both English and non-English languages. Through extensive pilot experiments, we empirically prove that both the multilingual capabilities and cultural adaptability of LLMs hold the potential to be significantly improved by XTransplant, respectively from En -> non-En and non-En -> En, highlighting the underutilization of current LLMs' multilingual potential. And the patterns observed in these pilot experiments further motivate an offline scaling inference strategy, which demonstrates consistent performance improvements in multilingual and culture-aware tasks, sometimes even surpassing multilingual supervised fine-tuning. And we do hope our further analysis and discussion could help gain deeper insights into XTransplant mechanism.

Related papers

• ShifCon: Enhancing Non-Dominant Language Capabilities with a Shift-based Contrastive Framework [78.07201802874529]
  ShifCon is a Shift-based Contrastive framework that aligns the internal forward process of other languages toward that of the dominant one. It shifts the representations of non-dominant languages into the dominant language subspace, allowing them to access relatively rich information encoded in the model parameters. Experiments demonstrate that our ShifCon framework significantly enhances the performance of non-dominant languages.
  arXiv  Detail & Related papers  (2024-10-25T10:28:59Z)
• Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models [104.96990850774566]
  We propose a Multi-lingual Ability Extraction and Transfer approach, named as MAET. Our key idea is to decompose and extract language-agnostic ability-related weights from large language models. Experiment results show MAET can effectively and efficiently extract and transfer the advanced abilities, and outperform training-based baseline methods.
  arXiv  Detail & Related papers  (2024-10-10T11:23:18Z)
• Probing the Emergence of Cross-lingual Alignment during LLM Training [10.053333786023089]
  Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We study how such cross-lingual alignment emerges during pre-training of LLMs. We observe a high correlation between neuron overlap and downstream performance.
  arXiv  Detail & Related papers  (2024-06-19T05:31:59Z)
• Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets [4.653113033432781]
  Cross-lingual transfer capabilities of Multilingual Language Models (MLLMs) are investigated. Our research provides valuable insights into cross-lingual transfer and its implications for NLP applications.
  arXiv  Detail & Related papers  (2024-03-29T08:47:15Z)
• Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning? [8.630930380973489]
  This paper investigates the potential benefits of employing machine translation as a continued training objective to enhance language representation learning. Our results show that, contrary to expectations, machine translation as the continued training fails to enhance cross-lingual representation learning. We conclude that explicit sentence-level alignment in the cross-lingual scenario is detrimental to cross-lingual transfer pretraining.
  arXiv  Detail & Related papers  (2024-03-25T13:53:04Z)
• Analysis of Multi-Source Language Training in Cross-Lingual Transfer [6.992785466925966]
  Cross-lingual transfer (XLT) methods have contributed to addressing this data scarcity problem. We show that the use of multiple source languages in XLT-a technique we term Multi-Source Language Training (MSLT)-leads to increased mingling of embedding spaces for different languages. On the other hand, we discover that using an arbitrary combination of source languages does not always guarantee better performance.
  arXiv  Detail & Related papers  (2024-02-21T06:37:07Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• High-resource Language-specific Training for Multilingual Neural Machine Translation [109.31892935605192]
  We propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder. HLT-MT is further trained on all available corpora to transfer knowledge from high-resource languages to low-resource languages.
  arXiv  Detail & Related papers  (2022-07-11T14:33:13Z)
• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision [42.724921817550516]
  We propose a network named decomposed attention (DA) as a replacement of MA. The DA consists of an intra-lingual attention (IA) and a cross-lingual attention (CA), which model intralingual and cross-lingual supervisions respectively. Experiments on various cross-lingual natural language understanding tasks show that the proposed architecture and learning strategy significantly improve the model's cross-lingual transferability.
  arXiv  Detail & Related papers  (2021-06-09T16:12:13Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.