Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts

• URL: http://arxiv.org/abs/2509.04982v1
• Date: Fri, 05 Sep 2025 10:08:14 GMT
• Title: Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts
• Authors: Julius Neumann, Robert Lange, Yuni Susanti, Michael Färber,
• Abstract summary: We evaluate the effectiveness of small Transformer-based models for sentiment classification in short texts.<n>We show that data augmentation improves classification performance, while continued pre-training on augmented datasets can introduce noise rather than boost accuracy.
• Score: 4.166512373146747
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sentiment classification in short text datasets faces significant challenges such as class imbalance, limited training samples, and the inherent subjectivity of sentiment labels -- issues that are further intensified by the limited context in short texts. These factors make it difficult to resolve ambiguity and exacerbate data sparsity, hindering effective learning. In this paper, we evaluate the effectiveness of small Transformer-based models (i.e., BERT and RoBERTa, with fewer than 1 billion parameters) for multi-label sentiment classification, with a particular focus on short-text settings. Specifically, we evaluated three key factors influencing model performance: (1) continued domain-specific pre-training, (2) data augmentation using automatically generated examples, specifically generative data augmentation, and (3) architectural variations of the classification head. Our experiment results show that data augmentation improves classification performance, while continued pre-training on augmented datasets can introduce noise rather than boost accuracy. Furthermore, we confirm that modifications to the classification head yield only marginal benefits. These findings provide practical guidance for optimizing BERT-based models in resource-constrained settings and refining strategies for sentiment classification in short-text datasets.

Related papers

• Based on Data Balancing and Model Improvement for Multi-Label Sentiment Classification Performance Enhancement [5.149011601951617]
  Multi-label sentiment classification plays a vital role in natural language processing by detecting multiple emotions within a single text.<n>Existing datasets like GoEmotions often suffer from severe class imbalance, which hampers model performance.<n>We constructed a balanced multi-label sentiment dataset using GoEmotions data, emotion-labeled samples from Sentiment140, and manually annotated texts.<n> Experimental results demonstrate significant improvements in accuracy, precision, recall, F1-score, and AUC compared to models trained on imbalanced data.
  arXiv  Detail & Related papers  (2025-11-18T03:06:27Z)
• Target-oriented Multimodal Sentiment Classification with Counterfactual-enhanced Debiasing [5.0175188046562385]
  multimodal sentiment classification seeks to predict sentiment polarity for specific targets from image-text pairs.<n>Existing works often over-rely on textual content and fail to consider dataset biases.<n>We introduce a novel counterfactual-enhanced debiasing framework to reduce such spurious correlations.
  arXiv  Detail & Related papers  (2025-09-11T05:40:53Z)
• Towards Robust Few-Shot Text Classification Using Transformer Architectures and Dual Loss Strategies [6.78820305740543]
  This paper proposes a strategy that combines adaptive fine-tuning, contrastive learning, and regularization optimization to improve the classification performance of Transformer-based models.<n>Experiments on the FewRel 2.0 dataset show that T5-small, DeBERTa-v3, and RoBERTa-base perform well in few-shot tasks.
  arXiv  Detail & Related papers  (2025-05-09T15:54:08Z)
• Batch Aggregation: An Approach to Enhance Text Classification with Correlated Augmented Data [0.0]
  We propose a novel approach called 'Batch Aggregation' (BAGG)<n>BAGG explicitly models the dependence of text inputs generated through augmentation by incorporating an additional layer that aggregates results from correlated texts.<n>We found that the increase of performance with BAGG is more obvious in domain specific data sets, with accuracy improvements of up to 10-29%.
  arXiv  Detail & Related papers  (2025-04-07T12:46:07Z)
• READ: Reinforcement-based Adversarial Learning for Text Classification with Limited Labeled Data [7.152603583363887]
  Pre-trained transformer models such as BERT have shown massive gains across many text classification tasks.<n>This paper proposes a method that encapsulates reinforcement learning-based text generation and semi-supervised adversarial learning approaches.<n>Our method READ, Reinforcement-based Adversarial learning, utilizes an unlabeled dataset to generate diverse synthetic text through reinforcement learning.
  arXiv  Detail & Related papers  (2025-01-14T11:39:55Z)
• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams [49.3179290313959]
  This study explores the efficacy of seven text sampling methods designed to selectively fine-tune language models. We precisely assess the impact of these methods on fine-tuning the SBERT model using four different loss functions. Our findings indicate that Softmax loss and Batch All Triplets loss are particularly effective for text stream classification.
  arXiv  Detail & Related papers  (2024-03-18T23:41:52Z)
• Text generation for dataset augmentation in security classification tasks [55.70844429868403]
  This study evaluates the application of natural language text generators to fill this data gap in multiple security-related text classification tasks. We find substantial benefits for GPT-3 data augmentation strategies in situations with severe limitations on known positive-class samples.
  arXiv  Detail & Related papers  (2023-10-22T22:25:14Z)
• Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties [62.997667081978825]
  In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class.
  arXiv  Detail & Related papers  (2021-12-15T18:56:39Z)
• Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification [59.698811329287174]
  We leverage GPT-2 for generating artificial training instances in order to improve classification performance. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements.
  arXiv  Detail & Related papers  (2021-11-17T12:10:03Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.