Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation

• URL: http://arxiv.org/abs/2605.01229v1
• Date: Sat, 02 May 2026 03:59:00 GMT
• Title: Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
• Abstract summary: Cross-attention patterns in neural machine translation are widely used to study how multilingual models align linguistic structure.<n>We report a systematic artifact in cross-attention analysis of NLLB-200 (600M)<n>We term these "attention sinks," extending findings from LLMs to NMT cross-attention and identifying a causal mechanism rooted in vocabulary design rather than position bias.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-attention patterns in neural machine translation (NMT) are widely used to study how multilingual models align linguistic structure. We report a systematic artifact in cross-attention analysis of NLLB-200 (600M): non-content tokens - primarily end-of-sequence tokens, language tags, and punctuation - capture 83 percent to 91 percent of total cross-attention mass. We term these "attention sinks," extending findings from LLMs [Xiao et al., 2023] to NMT cross-attention and identifying a causal mechanism rooted in vocabulary design rather than position bias. This artifact causes raw metrics to underestimate content-level similarity by nearly half (36.7 percent raw vs. 70.7 percent filtered), rendering uncorrected analyses unreliable. To address this, we validate a content-only filtering methodology that removes non-content tokens and renormalizes the distribution. Applying this to 1,000 parallel sentences across African languages (Swahili, Kikuyu, Somali, Luo) and non-African benchmarks (German, Turkish, Chinese, Hindi), we confirm the artifact is universal and recover masked linguistic signals: a 16.9 percentage-point gap between teacher-forcing and generation modes, clear language-family clustering in attention entropy, and a hidden Somali paradox linking SOV word order to monotonic alignment. We release our filtering toolkit and corrected datasets to support reproducible interpretability research on multilingual NMT.

Related papers

• BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translation [1.2546133965241097]
  BhashaSetu is a linguistically enriched English--Marathi parallel dataset addressing persistent data limitations in low-resource neural machine translation (NMT)<n>Our dataset comprises 2.78 million sentence pairs from heterogeneous sources including news, politics, healthcare, literature, and culture, with stemmed and lemmatized representations to support morphology-aware analysis.
  arXiv  Detail & Related papers  (2026-05-26T14:03:20Z)
• Language-Conditioned Visual Grounding with CLIP Multilingual [0.0]
  Cross-language divergence could arise from the visual encoder, the text branch, or their interaction.<n>We resolve this ambiguity through a dense multilingual CLIP probe in which the visual encoder is held identical across thirteen typologically diverse languages.
  arXiv  Detail & Related papers  (2026-05-09T17:06:16Z)
• MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes [0.0]
  This paper presents our system for the CHiPSAL 2026 shared task, addressing both Subtask A and Subtask B.<n>We propose a hybrid cross-modal attention fusion architecture that combines CLIP for visual encoding with BGE-M3 for multilingual text representation.<n>We show explicit cross-modal reasoning achieves a 5.9% F1-macro improvement over text-only baselines on Subtask A.
  arXiv  Detail & Related papers  (2026-04-13T07:37:14Z)
• DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects [6.107850985025956]
  We present DIA-HARM, the first benchmark for evaluating disinformation detection across 50 English dialects.<n>Human-written dialectal content degrades detection by 1.4-3.6% F1, while AI-generated content remains stable.<n>We release the DIA-HARM framework, D3 corpus, and evaluation tools.
  arXiv  Detail & Related papers  (2026-04-07T01:43:48Z)
• Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages [39.985923582735936]
  We introduce the FRED Difficulty Metrics, which include the Fertility Ratio (F), Retrieval Proxy (R), Pre-training Exposure (E) and Corpus Diversity (D)<n>These metrics reveal that a significant portion of result variability is explained by train-test overlap and pre-training exposure rather than model capability.
  arXiv  Detail & Related papers  (2026-03-26T09:20:17Z)
• Can We Still Hear the Accent? Investigating the Resilience of Native Language Signals in the LLM Era [0.0]
  We analyze native language identification trends in ACL Anthology papers across three eras: pre-neural network (NN), pre-LLM, and post-LLM.<n>Our analysis shows a consistent decline in NLI performance over time.<n>While Chinese and French show unexpected resistance or divergent trends, Japanese and Korean exhibit sharper-than-expected declines.
  arXiv  Detail & Related papers  (2026-03-20T09:11:18Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics [69.2321983942375]
  This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings.<n>We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (textitmatra) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi.<n>While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.
  arXiv  Detail & Related papers  (2026-02-19T14:56:42Z)
• Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levels [6.627057618324123]
  This paper introduces a fine-grained analysis of Whisper's multilingual decoder.<n>Our method traces the beam search path, capturing sub-token guesses and their associated probabilities.<n>Lower resource languages fare worse on these metrics, but also exhibit distinct clustering patterns in sub-token usage.
  arXiv  Detail & Related papers  (2025-09-29T21:20:05Z)
• Towards Faster k-Nearest-Neighbor Machine Translation [51.866464707284635]
  k-nearest-neighbor machine translation approaches suffer from heavy retrieve overhead on the entire datastore when decoding each token.<n>We propose a simple yet effective multi-layer perceptron (MLP) network to predict whether a token should be translated jointly by the neural machine translation model and probabilities produced by the kNN.<n>Our method significantly reduces the overhead of kNN retrievals by up to 53% at the expense of a slight decline in translation quality.
  arXiv  Detail & Related papers  (2023-12-12T16:41:29Z)
• Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding [53.84948040596055]
  We introduce two related methods to mitigate failure cases with a modified decoding objective. Experiments on the massively multilingual models M2M-100 (418M) and SMaLL-100 show that these methods suppress hallucinations and off-target translations.
  arXiv  Detail & Related papers  (2023-09-13T17:15:27Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.