Introducing Three New Benchmark Datasets for Hierarchical Text Classification

• URL: http://arxiv.org/abs/2411.19119v1
• Date: Thu, 28 Nov 2024 13:06:48 GMT
• Title: Introducing Three New Benchmark Datasets for Hierarchical Text Classification
• Authors: Jaco du Toit, Herman Redelinghuys, Marcel Dunaiski,
• Abstract summary: We introduce three new HTC benchmark datasets in the domain of research publications. We propose an approach which combines their classifications to improve the reliability and robustness of the dataset. We evaluate the three created datasets with a clustering-based analysis and show that our proposed approach results in a higher quality dataset.
• Score: 0.0
• License:
• Abstract: Hierarchical Text Classification (HTC) is a natural language processing task with the objective to classify text documents into a set of classes from a structured class hierarchy. Many HTC approaches have been proposed which attempt to leverage the class hierarchy information in various ways to improve classification performance. Machine learning-based classification approaches require large amounts of training data and are most-commonly compared through three established benchmark datasets, which include the Web Of Science (WOS), Reuters Corpus Volume 1 Version 2 (RCV1-V2) and New York Times (NYT) datasets. However, apart from the RCV1-V2 dataset which is well-documented, these datasets are not accompanied with detailed description methodologies. In this paper, we introduce three new HTC benchmark datasets in the domain of research publications which comprise the titles and abstracts of papers from the Web of Science publication database. We first create two baseline datasets which use existing journal-and citation-based classification schemas. Due to the respective shortcomings of these two existing schemas, we propose an approach which combines their classifications to improve the reliability and robustness of the dataset. We evaluate the three created datasets with a clustering-based analysis and show that our proposed approach results in a higher quality dataset where documents that belong to the same class are semantically more similar compared to the other datasets. Finally, we provide the classification performance of four state-of-the-art HTC approaches on these three new datasets to provide baselines for future studies on machine learning-based techniques for scientific publication classification.

Related papers

• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification [19.12354692458442]
  Hierarchical text classification (HTC) is a complex subtask under multi-label text classification. We propose HiGen, a text-generation-based framework utilizing language models to encode dynamic text representations.
  arXiv  Detail & Related papers  (2024-01-24T04:44:42Z)
• Generalized Category Discovery with Clustering Assignment Consistency [56.92546133591019]
  Generalized category discovery (GCD) is a recently proposed open-world task. We propose a co-training-based framework that encourages clustering consistency. Our method achieves state-of-the-art performance on three generic benchmarks and three fine-grained visual recognition datasets.
  arXiv  Detail & Related papers  (2023-10-30T00:32:47Z)
• ELFIS: Expert Learning for Fine-grained Image Recognition Using Subsets [6.632855264705276]
  We propose ELFIS, an expert learning framework for Fine-Grained Visual Recognition. A set of neural networks-based experts are trained focusing on the meta-categories and are integrated into a multi-task framework. Experiments show improvements in the SoTA FGVR benchmarks of up to +1.3% of accuracy using both CNNs and transformer-based networks.
  arXiv  Detail & Related papers  (2023-03-16T12:45:19Z)
• Simple multi-dataset detection [83.9604523643406]
  We present a simple method for training a unified detector on multiple large-scale datasets. We show how to automatically integrate dataset-specific outputs into a common semantic taxonomy. Our approach does not require manual taxonomy reconciliation.
  arXiv  Detail & Related papers  (2021-02-25T18:55:58Z)
• Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks [61.23408995934415]
  We propose a novel framework for minimally supervised categorization by learning from the text-rich network. Specifically, we jointly train two modules with different inductive biases -- a text analysis module for text understanding and a network learning module for class-discriminative, scalable network learning. Our experiments show that given only three seed documents per category, our framework can achieve an accuracy of about 92%.
  arXiv  Detail & Related papers  (2021-02-23T04:14:34Z)
• Multitask Learning for Class-Imbalanced Discourse Classification [74.41900374452472]
  We show that a multitask approach can improve 7% Micro F1-score upon current state-of-the-art benchmarks. We also offer a comparative review of additional techniques proposed to address resource-poor problems in NLP.
  arXiv  Detail & Related papers  (2021-01-02T07:13:41Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• GuCNet: A Guided Clustering-based Network for Improved Classification [15.747227188672088]
  We present a novel, and yet a very simple classification technique by leveraging the ease of classifiability of any existing well separable dataset for guidance. Since the guide dataset which may or may not have any semantic relationship with the experimental dataset, the proposed network tries to embed class-wise features of the challenging dataset to those distinct clusters of the guide set.
  arXiv  Detail & Related papers  (2020-10-11T10:22:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.