Cross-lingual Representation Learning via Centroid Intervention Fusion

• URL: http://arxiv.org/abs/2608.26357v1
• Date: Wed, 26 Aug 2026 19:39:57 GMT
• Title: Cross-lingual Representation Learning via Centroid Intervention Fusion
• Abstract summary: Centroid Intervention Fusion (CIF) is a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator.<n>CIF outperforms the strongest prior pairwise intervention baseline by up to +3.378 pp on average across four model backbones.
• Score: 30.50994950370956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) exhibit uneven multilingual performance, especially when dealing with low-resource languages. Inference-time intervention offers a lightweight way to improve cross-lingual transfer by modifying the hidden states produced by the LLMs during the forward pass, without updating model parameters. However, existing cross-lingual intervention methods typically learn separate projections from source to target languages, which limits scalability and prevents knowledge sharing across languages. We propose Centroid Intervention Fusion (CIF), a projection fusion framework that consolidates multiple multilingual intervention projections into a single language-shared operator. Across multilingual commonsense reasoning, natural language inference, factual editing, and machine translation benchmarks, CIF outperforms the strongest prior pairwise intervention baseline by up to +3.378 pp on average across four model backbones, while supporting performance gains for low resource languages. The code is available at https://github.com/VRCMF/CIF.git.

Related papers

• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation [50.731838216268706]
  We study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus.<n>We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers.<n>We propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers.
  arXiv  Detail & Related papers  (2026-05-28T10:12:41Z)
• COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling [0.023074632109535153]
  We introduce a data-centric framework for adapting large language models to target languages.<n>We use a distribution-aware sampling strategy to identify semantic gaps between existing training data and a target usage distribution.<n>We extend this into a continual learning framework, which monitors for data distribution shifts in production and dynamically updates adapters to prevent model staleness.
  arXiv  Detail & Related papers  (2026-04-22T16:07:10Z)
• Cross-Lingual Activation Steering for Multilingual Language Models [3.772378882850512]
  Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages.<n>We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates neuron activations.<n>Our results demonstrate that targeted activation steering can unlock latent multilingual capacity in existing models without modification to model weights.
  arXiv  Detail & Related papers  (2026-01-23T01:41:17Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.<n>But can these models relate corresponding concepts across languages, i.e., be crosslingual?<n>This study evaluates state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment [31.885608173448368]
  Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts. disparities in performance among different languages have been observed due to uneven resource availability. We propose a new parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities.
  arXiv  Detail & Related papers  (2023-05-02T14:09:02Z)
• Multilingual Entity and Relation Extraction from Unified to Language-specific Training [29.778332361215636]
  Existing approaches for entity and relation extraction tasks mainly focus on the English corpora and ignore other languages. We propose a two-stage multilingual training method and a joint model called Multilingual Entity and Relation Extraction framework (mERE) to mitigate language interference. Our method outperforms both the monolingual and multilingual baseline methods.
  arXiv  Detail & Related papers  (2023-01-11T12:26:53Z)
• A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning [6.329304732560936]
  Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries. We propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss.
  arXiv  Detail & Related papers  (2022-10-18T15:36:53Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.