Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling

• URL: http://arxiv.org/abs/2404.13078v2
• Date: Tue, 23 Apr 2024 05:15:18 GMT
• Title: Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling
• Authors: Darya Likhareva, Hamsini Sankaran, Sivakumar Thiyagarajan,
• Abstract summary: This paper introduces a novel approach using the SciBERT model and CNNs to systematically categorize academic abstracts. The CNN uses convolution and pooling to enhance feature extraction and reduce dimensionality.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Researchers must stay current in their fields by regularly reviewing academic literature, a task complicated by the daily publication of thousands of papers. Traditional multi-label text classification methods often ignore semantic relationships and fail to address the inherent class imbalances. This paper introduces a novel approach using the SciBERT model and CNNs to systematically categorize academic abstracts from the Elsevier OA CC-BY corpus. We use a multi-segment input strategy that processes abstracts, body text, titles, and keywords obtained via BERT topic modeling through SciBERT. Here, the [CLS] token embeddings capture the contextual representation of each segment, concatenated and processed through a CNN. The CNN uses convolution and pooling to enhance feature extraction and reduce dimensionality, optimizing the data for classification. Additionally, we incorporate class weights based on label frequency to address the class imbalance, significantly improving the classification F1 score and enhancing text classification systems and literature review efficiency.

Related papers

• Noise Contrastive Estimation-based Matching Framework for Low-Resource Security Attack Pattern Recognition [49.536368818512116]
  Tactics, Techniques and Procedures (TTPs) represent sophisticated attack patterns in the cybersecurity domain. We formulate the problem in a different learning paradigm, where the assignment of a text to a TTP label is decided by the direct semantic similarity between the two. We propose a neural matching architecture with an effective sampling-based learn-to-compare mechanism.
  arXiv  Detail & Related papers  (2024-01-18T19:02:00Z)
• A Process for Topic Modelling Via Word Embeddings [0.0]
  This work combines algorithms based on word embeddings, dimensionality reduction, and clustering. The objective is to obtain topics from a set of unclassified texts.
  arXiv  Detail & Related papers  (2023-10-06T15:10:35Z)
• A Visual Interpretation-Based Self-Improved Classification System Using Virtual Adversarial Training [4.722922834127293]
  This paper proposes a visual interpretation-based self-improving classification model with a combination of virtual adversarial training (VAT) and BERT models to address the problems. Specifically, a fine-tuned BERT model is used as a classifier to classify the sentiment of the text. The predicted sentiment classification labels are used as part of the input of another BERT for spam classification via a semi-supervised training manner.
  arXiv  Detail & Related papers  (2023-09-03T15:07:24Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text [1.1508304497344637]
  For large-scale IT corpora with hundreds of classes organized in a hierarchy, the task of accurate classification of classes at the higher level in the hierarchies is crucial. In the business world, an efficient and explainable ML model is preferred over an expensive black-box model, especially if the performance increase is marginal. Despite the widespread use of PLMs, there is a lack of a clear and well-justified need to as why these models are being employed for domain-specific text classification.
  arXiv  Detail & Related papers  (2023-03-31T03:17:23Z)
• SLCNN: Sentence-Level Convolutional Neural Network for Text Classification [0.0]
  Convolutional neural network (CNN) has shown remarkable success in the task of text classification. New baseline models have been studied for text classification using CNN. Results have shown that the proposed models have better performance, particularly in the longer documents.
  arXiv  Detail & Related papers  (2023-01-27T13:16:02Z)
• Many-Class Text Classification with Matching [65.74328417321738]
  We formulate textbfText textbfClassification as a textbfMatching problem between the text and the labels, and propose a simple yet effective framework named TCM. Compared with previous text classification approaches, TCM takes advantage of the fine-grained semantic information of the classification labels.
  arXiv  Detail & Related papers  (2022-05-23T15:51:19Z)
• Novel Class Discovery in Semantic Segmentation [104.30729847367104]
  We introduce a new setting of Novel Class Discovery in Semantic (NCDSS) It aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes. In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image. We propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels.
  arXiv  Detail & Related papers  (2021-12-03T13:31:59Z)
• Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification [60.233529926965836]
  We propose a new method called SHINE, which is based on graph neural network (GNN) for short text classification. First, we model the short text dataset as a hierarchical heterogeneous graph consisting of word-level component graphs. Then, we dynamically learn a short document graph that facilitates effective label propagation among similar short texts.
  arXiv  Detail & Related papers  (2021-10-30T05:33:05Z)
• ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models [0.0]
  Deep learning approaches based on CNN, LSTM, and Transformers have been the de facto approach for text classification. We show that these systems are over-reliant on the important words present in the text that are useful for classification.
  arXiv  Detail & Related papers  (2021-01-30T15:18:35Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.