Quantifying the Plausibility of Context Reliance in Neural Machine Translation

• URL: http://arxiv.org/abs/2310.01188v2
• Date: Wed, 13 Mar 2024 08:45:53 GMT
• Title: Quantifying the Plausibility of Context Reliance in Neural Machine Translation
• Authors: Gabriele Sarti, Grzegorz Chrupa{\l}a, Malvina Nissim, Arianna Bisazza
• Abstract summary: We introduce Plausibility Evaluation of Context Reliance (PECoRe) PECoRe is an end-to-end interpretability framework designed to quantify context usage in language models' generations. We use pecore to quantify the plausibility of context-aware machine translation models.
• Score: 25.29330352252055
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Establishing whether language models can use contextual information in a human-plausible way is important to ensure their trustworthiness in real-world settings. However, the questions of when and which parts of the context affect model generations are typically tackled separately, with current plausibility evaluations being practically limited to a handful of artificial benchmarks. To address this, we introduce Plausibility Evaluation of Context Reliance (PECoRe), an end-to-end interpretability framework designed to quantify context usage in language models' generations. Our approach leverages model internals to (i) contrastively identify context-sensitive target tokens in generated texts and (ii) link them to contextual cues justifying their prediction. We use \pecore to quantify the plausibility of context-aware machine translation models, comparing model rationales with human annotations across several discourse-level phenomena. Finally, we apply our method to unannotated model translations to identify context-mediated predictions and highlight instances of (im)plausible context usage throughout generation.

Related papers

• On Measuring Context Utilization in Document-Level MT Systems [12.02023514105999]
  We propose to complement accuracy-based evaluation with measures of context utilization. We show that automatically-annotated supporting context gives similar conclusions to human-annotated context.
  arXiv  Detail & Related papers  (2024-02-02T13:37:07Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [56.85319224208865]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling [0.0]
  The black-box structure of the deep neural network in pre-trained language models seriously limits the interpretability of the language modeling process. A Word-Context-Coupled Space (W2CSpace) is proposed by introducing the alignment processing between uninterpretable neural representation and interpretable statistical logic. Our language model can achieve better performance and highly credible interpretable ability compared to related state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-19T09:26:02Z)
• Reference-less Analysis of Context Specificity in Translation with Personalised Language Models [3.527589066359829]
  This work investigates what extent rich character and film annotations can be leveraged to personalise language models (LMs) We build LMs which leverage rich contextual information to reduce perplexity by up to 6.5% compared to a non-contextual model. Our results suggest that the degree to which professional translations in our domain are context-specific can be preserved to a better extent by a contextual machine translation model.
  arXiv  Detail & Related papers  (2023-03-29T12:19:23Z)
• HanoiT: Enhancing Context-aware Translation via Selective Context [95.93730812799798]
  Context-aware neural machine translation aims to use the document-level context to improve translation quality. The irrelevant or trivial words may bring some noise and distract the model from learning the relationship between the current sentence and the auxiliary context. We propose a novel end-to-end encoder-decoder model with a layer-wise selection mechanism to sift and refine the long document context.
  arXiv  Detail & Related papers  (2023-01-17T12:07:13Z)
• Contextual information integration for stance detection via cross-attention [59.662413798388485]
  Stance detection deals with identifying an author's stance towards a target. Most existing stance detection models are limited because they do not consider relevant contextual information. We propose an approach to integrate contextual information as text.
  arXiv  Detail & Related papers  (2022-11-03T15:04:29Z)
• Improve Discourse Dependency Parsing with Contextualized Representations [28.916249926065273]
  We propose to take advantage of transformers to encode contextualized representations of units of different levels. Motivated by the observation of writing patterns commonly shared across articles, we propose a novel method that treats discourse relation identification as a sequence labelling task.
  arXiv  Detail & Related papers  (2022-05-04T14:35:38Z)
• Measuring and Increasing Context Usage in Context-Aware Machine Translation [64.5726087590283]
  We introduce a new metric, conditional cross-mutual information, to quantify the usage of context by machine translation models. We then introduce a new, simple training method, context-aware word dropout, to increase the usage of context by context-aware models.
  arXiv  Detail & Related papers  (2021-05-07T19:55:35Z)
• How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context [59.13515950353125]
  We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-02-03T11:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.