CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification

• URL: http://arxiv.org/abs/2109.04853v1
• Date: Fri, 10 Sep 2021 13:09:12 GMT
• Title: CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification
• Authors: Mat\'u\v{s} Falis, Hang Dong, Alexandra Birch, Beatrice Alex
• Abstract summary: We argue for hierarchical evaluation of the predictions of neural LMTC models. We describe a structural issue in the representation of the structured label space in prior art. We propose a set of metrics for hierarchical evaluation using the depth-based representation.
• Score: 70.554573538777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries. Performance of models in prior art is evaluated with standard precision, recall, and F1 measures without regard for the rich hierarchical structure. In this work we argue for hierarchical evaluation of the predictions of neural LMTC models. With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology. We propose a set of metrics for hierarchical evaluation using the depth-based representation. We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III. We also propose further avenues of research involving the proposed ontological representation.

Related papers

• Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding [7.0413463890126735]
  This paper explores the use of Large Language Models (LLM), specifically the LLAMA architecture, to enhance ICD code classification. We evaluate these methods by comparing them against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-11-11T09:31:46Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Domain Adaptive Nuclei Instance Segmentation and Classification via Category-aware Feature Alignment and Pseudo-labelling [65.40672505658213]
  We propose a novel deep neural network, namely Category-Aware feature alignment and Pseudo-Labelling Network (CAPL-Net) for UDA nuclei instance segmentation and classification. Our approach outperforms state-of-the-art UDA methods with a remarkable margin.
  arXiv  Detail & Related papers  (2022-07-04T07:05:06Z)
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding [10.387366211090734]
  We propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable prediction of ICD codes from clinical documents. We evaluate HiLAT using hospital discharge summaries and their corresponding ICD-9 codes from the MIMIC-III database. Visualisations of attention weights present a potential explainability tool for checking the face validity of ICD code predictions.
  arXiv  Detail & Related papers  (2022-04-22T14:12:22Z)
• HFT-ONLSTM: Hierarchical and Fine-Tuning Multi-label Text Classification [7.176984223240199]
  Hierarchical multi-label text classification (HMTC) with higher accuracy over large sets of closely related categories has become a challenging problem. We present a hierarchical and fine-tuning approach based on the Ordered Neural LSTM neural network, abbreviated as HFT-ONLSTM, for more accurate level-by-level HMTC.
  arXiv  Detail & Related papers  (2022-04-18T00:57:46Z)
• Description-based Label Attention Classifier for Explainable ICD-9 Classification [10.407041139832955]
  We propose a description-based label attention classifier to improve the model explainability when dealing with noisy texts like clinical notes. We evaluate our proposed method with different transformer-based encoders on the MIMIC-III-50 dataset.
  arXiv  Detail & Related papers  (2021-09-24T15:31:38Z)
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction [64.42386426730695]
  International Classification of Diseases (ICD) are the de facto codes used globally for clinical coding. These codes enable healthcare providers to claim reimbursement and facilitate efficient storage and retrieval of diagnostic information. Our proposed approach enhances the performance of neural models by effectively training word vectors using routine medical data as well as external knowledge from scientific articles.
  arXiv  Detail & Related papers  (2021-02-26T17:49:58Z)
• Analysis and tuning of hierarchical topic models based on Renyi entropy approach [5.487882744996213]
  tuning of parameters of hierarchical models, including the number of topics on each hierarchical level, remains a challenging task. In this paper, we propose a Renyi entropy-based approach for a partial solution to the above problem.
  arXiv  Detail & Related papers  (2021-01-19T12:54:47Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Coherent Hierarchical Multi-Label Classification Networks [56.41950277906307]
  C-HMCNN(h) is a novel approach for HMC problems, which exploits hierarchy information in order to produce predictions coherent with the constraint and improve performance. We conduct an extensive experimental analysis showing the superior performance of C-HMCNN(h) when compared to state-of-the-art models.
  arXiv  Detail & Related papers  (2020-10-20T09:37:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.