Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation

• URL: http://arxiv.org/abs/2404.11201v1
• Date: Wed, 17 Apr 2024 09:33:19 GMT
• Title: Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation
• Abstract summary: Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. We propose Neuron, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks.
• Score: 7.376832526909754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. Language-specific modeling methods show promise in reducing interference. However, they often rely on heuristics to distribute capacity and struggle to foster cross-lingual transfer via isolated modules. In this paper, we explore intrinsic task modularity within multilingual networks and leverage these observations to circumvent interference under multilingual translation. We show that neurons in the feed-forward layers tend to be activated in a language-specific manner. Meanwhile, these specialized neurons exhibit structural overlaps that reflect language proximity, which progress across layers. Based on these findings, we propose Neuron Specialization, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks. Extensive experiments show that our approach achieves consistent performance gains over strong baselines with additional analyses demonstrating reduced interference and increased knowledge transfer.

Related papers

• When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training [57.230355403478995]
  We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM.<n>We find that shared concept spaces emerge early and continue to refine, but that alignment with them is language-dependent.<n>In contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior.
  arXiv  Detail & Related papers  (2026-01-30T11:23:01Z)
• The Transfer Neurons Hypothesis: An Underlying Mechanism for Language Latent Space Transitions in Multilingual LLMs [5.152664482521929]
  Recent studies have suggested a processing framework for multilingual inputs in decoder-based LLMs.<n>We show that certain neurons in the Neuron module are responsible for transferring representations between language-specific latent spaces and a shared semantic latent space.
  arXiv  Detail & Related papers  (2025-09-21T10:44:16Z)
• The Emergence of Abstract Thought in Large Language Models Beyond Any Language [95.50197866832772]
  Large language models (LLMs) function effectively across a diverse range of languages.<n>Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts.<n>Recent results show strong multilingual performance, even surpassing English performance on specific tasks in other languages.
  arXiv  Detail & Related papers  (2025-06-11T16:00:54Z)
• Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons [20.13484267765109]
  We study how multilingual language models evolve during pre-training.<n>We observe a transition from uniform language identification capabilities across layers to more specialized layer functions.<n>We identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.
  arXiv  Detail & Related papers  (2025-06-02T13:06:30Z)
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models [49.09746599881631]
  We present the first mechanistic interpretability study of language confusion.<n>We show that confusion points (CPs) are central to this phenomenon.<n>We show that editing a small set of critical neurons, identified via comparative analysis with multilingual-tuned models, substantially mitigates confusion.
  arXiv  Detail & Related papers  (2025-05-22T11:29:17Z)
• Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer [21.205821852762362]
  Existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of lowresource languages. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks.
  arXiv  Detail & Related papers  (2025-03-21T18:08:11Z)
• Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learning [69.8008228833895]
  We propose a small-sized generative neural network equipped with a continual learning mechanism. Our model prioritizes interpretability and demonstrates the advantages of online learning.
  arXiv  Detail & Related papers  (2024-12-23T10:23:47Z)
• XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation [49.69780199602105]
  Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability. We propose a probing method named XTransplant that explores cross-lingual latent interactions via cross-lingual feed-forward transplantation. We empirically prove that both the multilingual capabilities and cultural adaptability of LLMs hold the potential to be significantly improved by XTransplant.
  arXiv  Detail & Related papers  (2024-12-17T09:05:30Z)
• LANDeRMT: Detecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation [43.26446958873554]
  Large language models (LLMs) have shown promising results in multilingual translation even with limited bilingual supervision. Recent advancements in large language models (LLMs) have shown promising results in multilingual translation even with limited bilingual supervision. LandeRMT is a framework that selectively finetunes LLMs to textbfMachine textbfTranslation with diverse translation training data.
  arXiv  Detail & Related papers  (2024-09-29T02:39:42Z)
• Probing the Emergence of Cross-lingual Alignment during LLM Training [10.053333786023089]
  Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We study how such cross-lingual alignment emerges during pre-training of LLMs. We observe a high correlation between neuron overlap and downstream performance.
  arXiv  Detail & Related papers  (2024-06-19T05:31:59Z)
• Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs [70.3132264719438]
  We aim to fill the research gap by examining how neuron activation is shared across tasks and languages. We classify neurons into four distinct categories based on their responses to a specific input across different languages. Our analysis reveals the following insights: (i) the patterns of neuron sharing are significantly affected by the characteristics of tasks and examples; (ii) neuron sharing does not fully correspond with language similarity; (iii) shared neurons play a vital role in generating responses, especially those shared across all languages.
  arXiv  Detail & Related papers  (2024-06-13T16:04:11Z)
• Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks [15.758213562260913]
  We investigate the degree to which language-wise modularity naturally arises within models with no special modularity interventions. We show that language-specializedworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of Attributionworks in favor of more cross-lingual sharing.
  arXiv  Detail & Related papers  (2023-11-14T16:11:23Z)
• On the cross-lingual transferability of multilingual prototypical models across NLU tasks [2.44288434255221]
  Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications. In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages. This article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models.
  arXiv  Detail & Related papers  (2022-07-19T09:55:04Z)
• High-resource Language-specific Training for Multilingual Neural Machine Translation [109.31892935605192]
  We propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder. HLT-MT is further trained on all available corpora to transfer knowledge from high-resource languages to low-resource languages.
  arXiv  Detail & Related papers  (2022-07-11T14:33:13Z)
• Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance [55.10864476206503]
  We investigate the use of quantized vectors to model the latent linguistic embedding. By enforcing different policies over the latent spaces in the training, we are able to obtain a latent linguistic embedding. Our experiments show that the voice cloning system built with vector quantization has only a small degradation in terms of perceptive evaluations.
  arXiv  Detail & Related papers  (2021-06-25T07:51:35Z)
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment [59.995385574274785]
  We show that, contrary to previous belief, negative interference also impacts low-resource languages. We present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference.
  arXiv  Detail & Related papers  (2020-10-06T20:48:58Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.