Towards Robust Few-Shot Text Classification Using Transformer Architectures and Dual Loss Strategies

• URL: http://arxiv.org/abs/2505.06145v1
• Date: Fri, 09 May 2025 15:54:08 GMT
• Title: Towards Robust Few-Shot Text Classification Using Transformer Architectures and Dual Loss Strategies
• Abstract summary: This paper proposes a strategy that combines adaptive fine-tuning, contrastive learning, and regularization optimization to improve the classification performance of Transformer-based models.<n>Experiments on the FewRel 2.0 dataset show that T5-small, DeBERTa-v3, and RoBERTa-base perform well in few-shot tasks.
• Score: 6.78820305740543
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot text classification has important application value in low-resource environments. This paper proposes a strategy that combines adaptive fine-tuning, contrastive learning, and regularization optimization to improve the classification performance of Transformer-based models. Experiments on the FewRel 2.0 dataset show that T5-small, DeBERTa-v3, and RoBERTa-base perform well in few-shot tasks, especially in the 5-shot setting, which can more effectively capture text features and improve classification accuracy. The experiment also found that there are significant differences in the classification difficulty of different relationship categories. Some categories have fuzzy semantic boundaries or complex feature distributions, making it difficult for the standard cross entropy loss to learn the discriminative information required to distinguish categories. By introducing contrastive loss and regularization loss, the generalization ability of the model is enhanced, effectively alleviating the overfitting problem in few-shot environments. In addition, the research results show that the use of Transformer models or generative architectures with stronger self-attention mechanisms can help improve the stability and accuracy of few-shot classification.

Related papers

• Adapting Multimodal Foundation Models for Few-Shot Learning: A Comprehensive Study on Contrastive Captioners [1.2461503242570642]
  This paper presents a study on adapting the Contrastive Captioners (CoCa) visual backbone for few-shot image classification.<n>We identify an "augmentation divergence": while strong data augmentation degrades the performance of linear probing in low-shot settings, it is essential for stabilizing LoRA fine-tuning.<n>We also demonstrate that hybrid objectives incorporating Supervised Contrastive (SupCon) loss yield consistent performance improvements over standard Cross-Entropy.
  arXiv  Detail & Related papers  (2025-12-14T20:13:21Z)
• Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts [4.166512373146747]
  We evaluate the effectiveness of small Transformer-based models for sentiment classification in short texts.<n>We show that data augmentation improves classification performance, while continued pre-training on augmented datasets can introduce noise rather than boost accuracy.
  arXiv  Detail & Related papers  (2025-09-05T10:08:14Z)
• GLiClass: Generalist Lightweight Model for Sequence Classification Tasks [49.2639069781367]
  We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks.<n>Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios.
  arXiv  Detail & Related papers  (2025-08-11T06:22:25Z)
• Intelligently Augmented Contrastive Tensor Factorization: Empowering Multi-dimensional Time Series Classification in Low-Data Environments [4.77513566805416]
  We propose a versatile yet data-efficient framework, Intelligently Augmented Contrastive Factorization (ITA-CTF)<n>ITA-CTF module learns effective representations from multi-dimensional time series.<n>It incorporates a new contrastive loss optimization to similarity learning and class-awareness.<n>Compared to standard and several DL benchmarks, notable performance improvements up to 18.7% were achieved.
  arXiv  Detail & Related papers  (2025-05-03T11:28:13Z)
• Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer [0.0]
  This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks.<n>The improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate.
  arXiv  Detail & Related papers  (2025-01-23T08:32:27Z)
• Fine-grained Recognition with Learnable Semantic Data Augmentation [68.48892326854494]
  Fine-grained image recognition is a longstanding computer vision challenge. We propose diversifying the training data at the feature-level to alleviate the discriminative region loss problem. Our method significantly improves the generalization performance on several popular classification networks.
  arXiv  Detail & Related papers  (2023-09-01T11:15:50Z)
• Balanced Classification: A Unified Framework for Long-Tailed Object Detection [74.94216414011326]
  Conventional detectors suffer from performance degradation when dealing with long-tailed data due to a classification bias towards the majority head categories. We introduce a unified framework called BAlanced CLassification (BACL), which enables adaptive rectification of inequalities caused by disparities in category distribution. BACL consistently achieves performance improvements across various datasets with different backbones and architectures.
  arXiv  Detail & Related papers  (2023-08-04T09:11:07Z)
• TART: Improved Few-shot Text Classification Using Task-Adaptive Reference Transformation [23.02986307143718]
  We propose a novel Task-Adaptive Reference Transformation (TART) network to enhance the generalization. Our model surpasses the state-of-the-art method by 7.4% and 5.4% in 1-shot and 5-shot classification on the 20 Newsgroups dataset.
  arXiv  Detail & Related papers  (2023-06-03T18:38:02Z)
• Regularization Through Simultaneous Learning: A Case Study on Plant Classification [0.0]
  This paper introduces Simultaneous Learning, a regularization approach drawing on principles of Transfer Learning and Multi-task Learning. We leverage auxiliary datasets with the target dataset, the UFOP-HVD, to facilitate simultaneous classification guided by a customized loss function. Remarkably, our approach demonstrates superior performance over models without regularization.
  arXiv  Detail & Related papers  (2023-05-22T19:44:57Z)
• Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification [61.411869453639845]
  We introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. This design effectively helps the model to explore more subtle and discriminative features. Experimental results on three widely used fine-grained image classification datasets consistently show considerable improvements.
  arXiv  Detail & Related papers  (2022-11-30T16:55:14Z)
• Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classification [11.035878821365149]
  We propose a novel Adaptive Meta-learner via Gradient Similarity (AMGS) to improve the model generalization ability to a new task. Experimental results on several benchmarks demonstrate that the proposed AMGS consistently improves few-shot text classification performance.
  arXiv  Detail & Related papers  (2022-09-10T16:14:53Z)
• Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning [23.465747123791772]
  In few-shot classification, the primary goal is to learn representations that generalize well for novel classes. We propose an efficient low displacement rank (LDR) regularization strategy termed Ortho-Shot.
  arXiv  Detail & Related papers  (2021-10-18T14:58:36Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)
• Few-shot Classification via Adaptive Attention [93.06105498633492]
  We propose a novel few-shot learning method via optimizing and fast adapting the query sample representation based on very few reference samples. As demonstrated experimentally, the proposed model achieves state-of-the-art classification results on various benchmark few-shot classification and fine-grained recognition datasets.
  arXiv  Detail & Related papers  (2020-08-06T05:52:59Z)
• Understanding and Diagnosing Vulnerability under Adversarial Attacks [62.661498155101654]
  Deep Neural Networks (DNNs) are known to be vulnerable to adversarial attacks. We propose a novel interpretability method, InterpretGAN, to generate explanations for features used for classification in latent variables. We also design the first diagnostic method to quantify the vulnerability contributed by each layer.
  arXiv  Detail & Related papers  (2020-07-17T01:56:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.