Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning

• URL: http://arxiv.org/abs/2404.00224v1
• Date: Sat, 30 Mar 2024 02:52:14 GMT
• Title: Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning
• Authors: Gustavo Bartz Guedes, Ana Estela Antunes da Silva,
• Abstract summary: Our approach consists of fine-tuning transformer language models to generate sentence-level embeddings from scientific articles. We trained our models on three datasets with contrastive learning. We show that fine-tuning sentence transformers with contrastive learning and using the generated embeddings in downstream tasks is a feasible approach to sentence classification in scientific articles.
• Score: 1.104960878651584
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Scientific articles are long text documents organized into sections, each describing aspects of the research. Analyzing scientific production has become progressively challenging due to the increase in the number of available articles. Within this scenario, our approach consisted of fine-tuning transformer language models to generate sentence-level embeddings from scientific articles, considering the following labels: background, objective, methods, results, and conclusion. We trained our models on three datasets with contrastive learning. Two datasets are from the article's abstracts in the computer science and medical domains. Also, we introduce PMC-Sents-FULL, a novel dataset of sentences extracted from the full texts of medical articles. We compare the fine-tuned and baseline models in clustering and classification tasks to evaluate our approach. On average, clustering agreement measures values were five times higher. For the classification measures, in the best-case scenario, we had an average improvement in F1-micro of 30.73\%. Results show that fine-tuning sentence transformers with contrastive learning and using the generated embeddings in downstream tasks is a feasible approach to sentence classification in scientific articles. Our experiment codes are available on GitHub.

Related papers

• Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes [44.51779041553597]
  FusionSent is an efficient and prompt-free approach for few-shot classification of scientific documents with many classes. Experiments show that FusionSent significantly outperforms strong baselines by an average of $6.0$ $F_1$label points.
  arXiv  Detail & Related papers  (2024-10-08T07:52:35Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification [0.27195102129095]
  We show that classification tasks still suffer from a performance gap when the underlying distribution of topics changes. We quantify this phenomenon empirically with a large corpus and a large set of topics. We suggest and successfully test a possible remedy: after augmenting the training dataset with topically-controlled synthetic texts, the F1 score improves by up to 50% for some topics.
  arXiv  Detail & Related papers  (2023-11-27T18:53:31Z)
• RankAug: Augmented data ranking for text classification [0.0]
  RankAug is a text-ranking approach that detects and filters out the top augmented texts. We demonstrate that the judicious selection of filtering techniques can yield a substantial improvement of up to 35% in classification accuracy for under-represented classes.
  arXiv  Detail & Related papers  (2023-11-08T08:47:49Z)
• MuLMS-AZ: An Argumentative Zoning Dataset for the Materials Science Domain [1.209268134212644]
  Classifying the Argumentative Zone (AZ) has been proposed to improve processing of scholarly documents. We present and release a new dataset of 50 manually annotated research articles.
  arXiv  Detail & Related papers  (2023-07-05T14:55:18Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework [68.04940365847543]
  We propose a semi-supervised sentence embedding framework, GenSE, that effectively leverages large-scale unlabeled data. Our method include three parts: 1) Generate: A generator/discriminator model is jointly trained to synthesize sentence pairs from open-domain unlabeled corpus; 2) Discriminate: Noisy sentence pairs are filtered out by the discriminator to acquire high-quality positive and negative sentence pairs; 3) Contrast: A prompt-based contrastive approach is presented for sentence representation learning with both annotated and synthesized data.
  arXiv  Detail & Related papers  (2022-10-30T10:15:21Z)
• Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation [81.55533657694016]
  We propose a Hierarchical bi-directional self-attention Network framework (HabNet) for paper review rating prediction and recommendation. Specifically, we leverage the hierarchical structure of the paper reviews with three levels of encoders: sentence encoder (level one), intra-review encoder (level two) and inter-review encoder (level three) We are able to identify useful predictors to make the final acceptance decision, as well as to help discover the inconsistency between numerical review ratings and text sentiment conveyed by reviewers.
  arXiv  Detail & Related papers  (2020-11-02T08:07:50Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)
• A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
  The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
  arXiv  Detail & Related papers  (2020-08-02T00:09:03Z)
• Segmenting Scientific Abstracts into Discourse Categories: A Deep Learning-Based Approach for Sparse Labeled Data [8.635930195821265]
  We train a deep neural network on structured abstracts from PubMed to fine-tune it on a small hand-labeled corpus of computer science papers. Our method appears to be a promising solution to the automatic segmentation of abstracts, where the data is sparse.
  arXiv  Detail & Related papers  (2020-05-11T20:21:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.