Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining

• URL: http://arxiv.org/abs/2308.03235v1
• Date: Mon, 7 Aug 2023 01:10:50 GMT
• Title: Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• Authors: Nour Eddine Zekaoui, Siham Yousfi, Maryem Rhanoui, Mounia Mikram
• Abstract summary: This paper studies the behaviour of the cutting-edge Transformer-based language models on opinion mining. Our comparative study shows leads and paves the way for production engineers regarding the approach to focus on.
• Score: 0.5735035463793008
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Opinion mining, also known as sentiment analysis, is a subfield of natural language processing (NLP) that focuses on identifying and extracting subjective information in textual material. This can include determining the overall sentiment of a piece of text (e.g., positive or negative), as well as identifying specific emotions or opinions expressed in the text, that involves the use of advanced machine and deep learning techniques. Recently, transformer-based language models make this task of human emotion analysis intuitive, thanks to the attention mechanism and parallel computation. These advantages make such models very powerful on linguistic tasks, unlike recurrent neural networks that spend a lot of time on sequential processing, making them prone to fail when it comes to processing long text. The scope of our paper aims to study the behaviour of the cutting-edge Transformer-based language models on opinion mining and provide a high-level comparison between them to highlight their key particularities. Additionally, our comparative study shows leads and paves the way for production engineers regarding the approach to focus on and is useful for researchers as it provides guidelines for future research subjects.

Related papers

• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• Is English the New Programming Language? How About Pseudo-code Engineering? [0.0]
  This study investigates how different input forms impact ChatGPT, a leading language model by OpenAI. It examines the model's proficiency across four categories: understanding of intentions, interpretability, completeness, and creativity.
  arXiv  Detail & Related papers  (2024-04-08T16:28:52Z)
• Detection of Machine-Generated Text: Literature Survey [0.0]
  This literature survey aims to compile and synthesize accomplishments and developments in the field of machine-generated text. It also gives an overview of machine-generated text trends and explores the larger societal implications.
  arXiv  Detail & Related papers  (2024-01-02T01:44:15Z)
• Picking the Underused Heads: A Network Pruning Perspective of Attention Head Selection for Fusing Dialogue Coreference Information [50.41829484199252]
  Transformer-based models with the multi-head self-attention mechanism are widely used in natural language processing. We investigate the attention head selection and manipulation strategy for feature injection from a network pruning perspective.
  arXiv  Detail & Related papers  (2023-12-15T05:27:24Z)
• Beyond Turing: A Comparative Analysis of Approaches for Detecting Machine-Generated Text [1.919654267936118]
  Traditional shallow learning, Language Model (LM) fine-tuning, and Multilingual Model fine-tuning are evaluated. Results reveal considerable differences in performance across methods. This study paves the way for future research aimed at creating robust and highly discriminative models.
  arXiv  Detail & Related papers  (2023-11-21T06:23:38Z)
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks [60.38369406877899]
  Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. Our survey encompasses the identification of the top five application domains for transformer-based models.
  arXiv  Detail & Related papers  (2023-06-11T23:13:51Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [56.85319224208865]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• REDAffectiveLM: Leveraging Affect Enriched Embedding and Transformer-based Neural Language Model for Readers' Emotion Detection [3.6678641723285446]
  We propose a novel approach for Readers' Emotion Detection from short-text documents using a deep learning model called REDAffectiveLM. We leverage context-specific and affect enriched representations by using a transformer-based pre-trained language model in tandem with affect enriched Bi-LSTM+Attention.
  arXiv  Detail & Related papers  (2023-01-21T19:28:25Z)
• Probing Speech Emotion Recognition Transformers for Linguistic Knowledge [7.81884995637243]
  We investigate the extent in which linguistic information is exploited during speech emotion recognition fine-tuning. We synthesise prosodically neutral speech utterances while varying the sentiment of the text. Valence predictions of the transformer model are very reactive to positive and negative sentiment content, as well as negations, but not to intensifiers or reducers.
  arXiv  Detail & Related papers  (2022-04-01T12:47:45Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey [54.34370423151014]
  This paper surveys the components of modeling approaches relaying task impacts across various generation tasks such as storytelling, summarization, translation etc. We present an abstraction of the imperative techniques with respect to learning paradigms, pretraining, modeling approaches, decoding and the key challenges outstanding in the field in each of them.
  arXiv  Detail & Related papers  (2020-10-14T17:54:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.