Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer

• URL: http://arxiv.org/abs/2501.13467v1
• Date: Thu, 23 Jan 2025 08:32:27 GMT
• Title: Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer
• Authors: Jia Gao, Guiran Liu, Binrong Zhu, Shicheng Zhou, Hongye Zheng, Xiaoxuan Liao,
• Abstract summary: This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks. The improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate.
• Score: 0.0
• License:
• Abstract: This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks. Aiming at the shortcomings of the traditional Transformer model in capturing deep semantic relationships and optimizing computational complexity, this paper introduces a multi-level attention mechanism and a contrastive learning strategy. The multi-level attention mechanism effectively models the global semantics and local features in the text by combining global attention with local attention; the contrastive learning strategy enhances the model's ability to distinguish between different categories by constructing positive and negative sample pairs while improving the classification effect. In addition, in order to improve the training and inference efficiency of the model on large-scale text data, this paper designs a lightweight module to optimize the feature transformation process and reduce the computational cost. Experimental results on the dataset show that the improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate, showing stronger semantic representation ability and generalization performance. The method proposed in this paper provides a new idea for algorithm optimization in the field of text classification and has good application potential and practical value. Future work will focus on studying the performance of this model in multi-category imbalanced datasets and cross-domain tasks and explore the integration wi

Related papers

• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• Research on Named Entity Recognition in Improved transformer with R-Drop structure [3.677017987610888]
  The XLNet-Transformer-R model is proposed in this paper. The Transformer encoder with relative positional encodings are combined to enhance the model's ability to process long text. To prevent overfitting, the R-Drop structure is used to improve the generalization capability.
  arXiv  Detail & Related papers  (2023-06-14T07:34:27Z)
• Adversarial Adaptation for French Named Entity Recognition [21.036698406367115]
  We propose a Transformer-based NER approach for French, using adversarial adaptation to similar domain or general corpora. Our approach allows learning better features using large-scale unlabeled corpora from the same domain or mixed domains. We also show that adversarial adaptation to large-scale unlabeled corpora can help mitigate the performance dip incurred on using Transformer models pre-trained on smaller corpora.
  arXiv  Detail & Related papers  (2023-01-12T18:58:36Z)
• Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classification [11.035878821365149]
  We propose a novel Adaptive Meta-learner via Gradient Similarity (AMGS) to improve the model generalization ability to a new task. Experimental results on several benchmarks demonstrate that the proposed AMGS consistently improves few-shot text classification performance.
  arXiv  Detail & Related papers  (2022-09-10T16:14:53Z)
• Pre-Training a Graph Recurrent Network for Language Representation [34.4554387894105]
  We consider a graph recurrent network for language model pre-training, which builds a graph structure for each sequence with local token-level communications. We find that our model can generate more diverse outputs with less contextualized feature redundancy than existing attention-based models.
  arXiv  Detail & Related papers  (2022-09-08T14:12:15Z)
• Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification [59.698811329287174]
  We leverage GPT-2 for generating artificial training instances in order to improve classification performance. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements.
  arXiv  Detail & Related papers  (2021-11-17T12:10:03Z)
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures [57.46093180685175]
  We demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. We add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales.
  arXiv  Detail & Related papers  (2021-06-10T15:41:53Z)
• On Learning Text Style Transfer with Direct Rewards [101.97136885111037]
  Lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task. We leverage semantic similarity metrics originally used for fine-tuning neural machine translation models. Our model provides significant gains in both automatic and human evaluation over strong baselines.
  arXiv  Detail & Related papers  (2020-10-24T04:30:02Z)
• Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks [75.69896269357005]
  Mixup is the latest data augmentation technique that linearly interpolates input examples and the corresponding labels. In this paper, we explore how to apply mixup to natural language processing tasks. We incorporate mixup to transformer-based pre-trained architecture, named "mixup-transformer", for a wide range of NLP tasks.
  arXiv  Detail & Related papers  (2020-10-05T23:37:30Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)
• Incorporating Effective Global Information via Adaptive Gate Attention for Text Classification [13.45504908358177]
  We show that simple statistical information can enhance classification performance both efficiently and significantly compared with several baseline models. We propose a classifier with gate mechanism named Adaptive Gate Attention model with Global Information (AGA+GI) in which the adaptive gate mechanism incorporates global statistical features into latent semantic features. Our experiments show that the proposed method can achieve better accuracy than CNN-based and RNN-based approaches without global information on several benchmarks.
  arXiv  Detail & Related papers  (2020-02-22T10:06:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.