Advancing Explainability in Neural Machine Translation: Analytical Metrics for Attention and Alignment Consistency

• URL: http://arxiv.org/abs/2412.18669v1
• Date: Tue, 24 Dec 2024 20:08:33 GMT
• Title: Advancing Explainability in Neural Machine Translation: Analytical Metrics for Attention and Alignment Consistency
• Authors: Anurag Mishra,
• Abstract summary: We introduce a systematic framework to quantitatively evaluate the explainability of an NMT model attention patterns. We present a set of metrics attention entropy and alignment agreement and validate them on an English-German test subset from WMT14. Our results indicate that sharper attention distributions correlate with improved interpretability but do not always guarantee better translation quality.
• Score: 2.4022340214033915
• License:
• Abstract: Neural Machine Translation (NMT) models have shown remarkable performance but remain largely opaque in their decision making processes. The interpretability of these models, especially their internal attention mechanisms, is critical for building trust and verifying that these systems behave as intended. In this work, we introduce a systematic framework to quantitatively evaluate the explainability of an NMT model attention patterns by comparing them against statistical alignments and correlating them with standard machine translation quality metrics. We present a set of metrics attention entropy and alignment agreement and validate them on an English-German test subset from WMT14 using a pre trained mT5 model. Our results indicate that sharper attention distributions correlate with improved interpretability but do not always guarantee better translation quality. These findings advance our understanding of NMT explainability and guide future efforts toward building more transparent and reliable machine translation systems.

Related papers

• Visualizing Uncertainty in Translation Tasks: An Evaluation of LLM Performance and Confidence Metrics [0.20971479389679337]
  Large language models (LLMs) are increasingly utilized for machine translation, yet their predictions often exhibit uncertainties that hinder interpretability and user trust. This paper addresses two primary objectives: (1) providing users with token-level insights into model confidence and (2) developing a web-based visualization tool to quantify and represent translation uncertainties.
  arXiv  Detail & Related papers  (2025-01-26T17:14:51Z)
• Context-Aware Machine Translation with Source Coreference Explanation [26.336947440529713]
  We propose a model that explains the decisions made for translation by predicting coreference features in the input. We evaluate our method in the WMT document-level translation task of English-German dataset, the English-Russian dataset, and the multilingual TED talk dataset.
  arXiv  Detail & Related papers  (2024-04-30T12:41:00Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality? [61.866103154161884]
  Neural machine translation (NMT) is often criticized for failures that happen without awareness. We propose a novel competency-aware NMT by extending conventional NMT with a self-estimator. We show that the proposed method delivers outstanding performance on quality estimation.
  arXiv  Detail & Related papers  (2022-11-25T02:39:41Z)
• Understanding and Mitigating the Uncertainty in Zero-Shot Translation [92.25357943169601]
  We aim to understand and alleviate the off-target issues from the perspective of uncertainty in zero-shot translation. We propose two lightweight and complementary approaches to denoise the training data for model training. Our approaches significantly improve the performance of zero-shot translation over strong MNMT baselines.
  arXiv  Detail & Related papers  (2022-05-20T10:29:46Z)
• Supervised Visual Attention for Simultaneous Multimodal Machine Translation [47.18251159303909]
  We propose the first Transformer-based simultaneous machine translation (MMT) architecture. We extend this model with an auxiliary supervision signal that guides its visual attention mechanism using labelled phrase-region alignments. Our results show that supervised visual attention consistently improves the translation quality of the MMT models.
  arXiv  Detail & Related papers  (2022-01-23T17:25:57Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents [40.25277134147149]
  We present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation. Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.
  arXiv  Detail & Related papers  (2020-11-04T04:44:47Z)
• On the Inference Calibration of Neural Machine Translation [54.48932804996506]
  We study the correlation between calibration and translation performance and linguistic properties of miscalibration. We propose a new graduated label smoothing method that can improve both inference calibration and translation performance.
  arXiv  Detail & Related papers  (2020-05-03T02:03:56Z)
• Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation [7.993547048820065]
  We introduce the first of their kind MT benchmark datasets that aim to track and hail improvements across four main discourse phenomena. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena.
  arXiv  Detail & Related papers  (2020-04-30T07:15:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.