Linguistic Fingerprint in Transformer Models: How Language Variation Influences Parameter Selection in Irony Detection

• URL: http://arxiv.org/abs/2406.02338v1
• Date: Tue, 4 Jun 2024 14:09:36 GMT
• Title: Linguistic Fingerprint in Transformer Models: How Language Variation Influences Parameter Selection in Irony Detection
• Authors: Michele Mastromattei, Fabio Massimo Zanzotto,
• Abstract summary: We aim to investigate how different English variations impact transformer-based models for irony detection. Our results reveal several similarities between optimalworks, which provide insights into the linguistic variations that share strong resemblances and those that exhibit greater dissimilarities. This study highlights the inherent structural similarities between models trained on different variants of the same language and also the critical role of parameter values in capturing these nuances.
• Score: 1.5807079236265718
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: This paper explores the correlation between linguistic diversity, sentiment analysis and transformer model architectures. We aim to investigate how different English variations impact transformer-based models for irony detection. To conduct our study, we used the EPIC corpus to extract five diverse English variation-specific datasets and applied the KEN pruning algorithm on five different architectures. Our results reveal several similarities between optimal subnetworks, which provide insights into the linguistic variations that share strong resemblances and those that exhibit greater dissimilarities. We discovered that optimal subnetworks across models share at least 60% of their parameters, emphasizing the significance of parameter values in capturing and interpreting linguistic variations. This study highlights the inherent structural similarities between models trained on different variants of the same language and also the critical role of parameter values in capturing these nuances.

Related papers

• A Joint Matrix Factorization Analysis of Multilingual Representations [28.751144371901958]
  We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. We study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models.
  arXiv  Detail & Related papers  (2023-10-24T04:43:45Z)
• Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing [2.5002227227256864]
  We present experiments with semantic structural probing, a method for studying sentence-level representations. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.
  arXiv  Detail & Related papers  (2023-10-18T12:32:07Z)
• Semantic Change Detection for the Romanian Language [0.5202524136984541]
  We analyze different strategies to create static and contextual word embedding models on real-world datasets. We first evaluate both word embedding models on an English dataset (SEMEVAL-CCOHA) and then on a Romanian dataset. The experimental results show that, depending on the corpus, the most important factors to consider are the choice of model and the distance to calculate a score for detecting semantic change.
  arXiv  Detail & Related papers  (2023-08-23T13:37:02Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Corpus-Guided Contrast Sets for Morphosyntactic Feature Detection in Low-Resource English Varieties [3.3536302616846734]
  We present a human-in-the-loop approach to generate and filter effective contrast sets via corpus-guided edits. We show that our approach improves feature detection for both Indian English and African American English, demonstrate how it can assist linguistic research, and release our fine-tuned models for use by other researchers.
  arXiv  Detail & Related papers  (2022-09-15T21:19:31Z)
• Visual Comparison of Language Model Adaptation [55.92129223662381]
  adapters are lightweight alternatives for model adaptation. In this paper, we discuss several design and alternatives for interactive, comparative visual explanation methods. We show that, for instance, an adapter trained on the language debiasing task according to context-0 embeddings introduces a new type of bias.
  arXiv  Detail & Related papers  (2022-08-17T09:25:28Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Parameter Differentiation based Multilingual Neural Machine Translation [37.16691633466614]
  Multilingual neural machine translation (MNMT) aims to translate multiple languages with a single model. It is still an open question which parameters should be shared and which ones need to be task-specific. We propose a novel parameter differentiation based method that allows the model to determine which parameters should be language-specific.
  arXiv  Detail & Related papers  (2021-12-27T11:41:52Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Composed Variational Natural Language Generation for Few-shot Intents [118.37774762596123]
  We generate training examples for few-shot intents in the realistic imbalanced scenario. To evaluate the quality of the generated utterances, experiments are conducted on the generalized few-shot intent detection task. Our proposed model achieves state-of-the-art performances on two real-world intent detection datasets.
  arXiv  Detail & Related papers  (2020-09-21T17:48:43Z)
• Evaluating Transformer-Based Multilingual Text Classification [55.53547556060537]
  We argue that NLP tools perform unequally across languages with different syntactic and morphological structures. We calculate word order and morphological similarity indices to aid our empirical study.
  arXiv  Detail & Related papers  (2020-04-29T03:34:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.