Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

• URL: http://arxiv.org/abs/2403.12809v1
• Date: Tue, 19 Mar 2024 15:07:22 GMT
• Title: Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models
• Authors: Zhixue Zhao, Nikolaos Aletras,
• Abstract summary: We show that FA faithfulness varies between multilingual and monolingual models. The larger the multilingual model, the less faithful the FAs are compared to its counterpart monolingual models. Our further analysis shows that the faithfulness disparity is potentially driven by the differences between model tokenizers.
• Score: 39.503228364872434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature attribution methods (FAs) provide insights into how different parts of the input contribute to a prediction. Previous studies have explored how different factors affect faithfulness, mainly in the context of monolingual English models. On the other hand, the differences in FA faithfulness between multilingual and monolingual models have yet to be explored. Our extensive experiments, covering five languages and five popular FAs, show that FA faithfulness varies between multilingual and monolingual models. We find that the larger the multilingual model, the less faithful the FAs are compared to its counterpart monolingual models.Our further analysis shows that the faithfulness disparity is potentially driven by the differences between model tokenizers. Our code is available: https://github.com/casszhao/multilingual-faith.

Related papers

• Do Multilingual Large Language Models Mitigate Stereotype Bias? [9.31741279000585]
  This study systematically trains six LLMs of identical size and architecture in English, German, French, Italian, and Spanish. We observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models.
  arXiv  Detail & Related papers  (2024-07-08T08:46:50Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do. We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models. Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning [44.53966523376327]
  SeaEval is a benchmark for multilingual foundation models. We characterize how these models understand and reason with natural language. We also investigate how well they comprehend cultural practices, nuances, and values.
  arXiv  Detail & Related papers  (2023-09-09T11:42:22Z)
• Are Pretrained Multilingual Models Equally Fair Across Languages? [0.0]
  This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. We evaluate three multilingual models on MozArt -- mBERT, XLM-R, and mT5 -- and show that across the four target languages, the three models exhibit different levels of group disparity.
  arXiv  Detail & Related papers  (2022-10-11T13:59:19Z)
• MonoByte: A Pool of Monolingual Byte-level Language Models [4.491765479948667]
  We release 10 monolingual byte-level models rigorously pretrained under the same configuration. Because they are tokenizer-free, the problem of unseen token embeddings is eliminated. Experiments on QA and NLI tasks show that our monolingual models achieve competitive performance to the multilingual one.
  arXiv  Detail & Related papers  (2022-09-22T14:32:48Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models [96.32118305166412]
  We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We find that languages which are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts.
  arXiv  Detail & Related papers  (2020-12-31T14:11:00Z)
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling [73.40368222437912]
  We propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models to the unified multilingual model (student) Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
  arXiv  Detail & Related papers  (2020-04-08T07:14:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.