Comprehensive Comparative Study of Multi-Label Classification Methods

• URL: http://arxiv.org/abs/2102.07113v2
• Date: Tue, 16 Feb 2021 05:29:43 GMT
• Title: Comprehensive Comparative Study of Multi-Label Classification Methods
• Authors: Jasmin Bogatinovski, Ljup\v{c}o Todorovski, Sa\v{s}o D\v{z}eroski, Dragi Kocev
• Abstract summary: Multi-label classification (MLC) has recently received increasing interest from the machine learning community. This work provides a comprehensive empirical study of a wide range of MLC methods on a plethora of datasets from various domains.
• Score: 1.1278903078792917
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-label classification (MLC) has recently received increasing interest from the machine learning community. Several studies provide reviews of methods and datasets for MLC and a few provide empirical comparisons of MLC methods. However, they are limited in the number of methods and datasets considered. This work provides a comprehensive empirical study of a wide range of MLC methods on a plethora of datasets from various domains. More specifically, our study evaluates 26 methods on 42 benchmark datasets using 20 evaluation measures. The adopted evaluation methodology adheres to the highest literature standards for designing and executing large scale, time-budgeted experimental studies. First, the methods are selected based on their usage by the community, assuring representation of methods across the MLC taxonomy of methods and different base learners. Second, the datasets cover a wide range of complexity and domains of application. The selected evaluation measures assess the predictive performance and the efficiency of the methods. The results of the analysis identify RFPCT, RFDTBR, ECCJ48, EBRJ48 and AdaBoostMH as best performing methods across the spectrum of performance measures. Whenever a new method is introduced, it should be compared to different subsets of MLC methods, determined on the basis of the different evaluation criteria.

Related papers

• Generating Synthetic Electronic Health Record (EHR) Data: A Review with Benchmarking [16.827140823757603]
  Methods are benchmarked on open-source EHR datasets, MIMIC-III/IV. We provide a decision tree to guide the choice among the benchmarked methods. GAN-based methods excel when distributional shifts exist between the training and testing populations.
  arXiv  Detail & Related papers  (2024-11-06T21:59:19Z)
• POGEMA: A Benchmark Platform for Cooperative Multi-Agent Navigation [76.67608003501479]
  We introduce and specify an evaluation protocol defining a range of domain-related metrics computed on the basics of the primary evaluation indicators. The results of such a comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.
  arXiv  Detail & Related papers  (2024-07-20T16:37:21Z)
• Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification [4.588028371034407]
  We focus on analyzing InterpretTime, a recent evaluation methodology for attribution methods applied to MTSC. We showcase some significant weaknesses of the original methodology and propose ideas to improve its accuracy and efficiency. We find that perturbation-based methods such as SHAP and Feature Ablation work well across a set of datasets.
  arXiv  Detail & Related papers  (2024-06-18T11:18:46Z)
• Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement [69.51496713076253]
  In this paper, we focus on the aforementioned efficiency aspects of existing MTL methods. We first carry out large-scale experiments of the methods with smaller backbones and on a the MetaGraspNet dataset as a new test ground. We also propose Feature Disentanglement measure as a novel and efficient identifier of the challenges in MTL.
  arXiv  Detail & Related papers  (2024-02-05T22:15:55Z)
• Current Methods for Drug Property Prediction in the Real World [9.061842820405486]
  Predicting drug properties is key in drug discovery to enable de-risking of assets before expensive clinical trials. It remains unclear for practitioners which method or approach is most suitable, as different papers benchmark on different datasets and methods. Our large-scale empirical study links together numerous earlier works on different datasets and methods. We discover that the best method depends on the dataset, and that engineered features with classical ML methods often outperform deep learning.
  arXiv  Detail & Related papers  (2023-07-25T17:50:05Z)
• Better Understanding Differences in Attribution Methods via Systematic Evaluations [57.35035463793008]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models.
  arXiv  Detail & Related papers  (2023-03-21T14:24:58Z)
• A One-shot Framework for Distributed Clustered Learning in Heterogeneous Environments [54.172993875654015]
  The paper proposes a family of communication efficient methods for distributed learning in heterogeneous environments. One-shot approach, based on local computations at the users and a clustering based aggregation step at the server is shown to provide strong learning guarantees. For strongly convex problems it is shown that, as long as the number of data points per user is above a threshold, the proposed approach achieves order-optimal mean-squared error rates in terms of the sample size.
  arXiv  Detail & Related papers  (2022-09-22T09:04:10Z)
• On the role of benchmarking data sets and simulations in method comparison studies [0.0]
  This paper investigates differences and similarities between simulation studies and benchmarking studies. We borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.
  arXiv  Detail & Related papers  (2022-08-02T13:47:53Z)
• Towards Better Understanding Attribution Methods [77.1487219861185]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We also propose a post-processing smoothing step that significantly improves the performance of some attribution methods.
  arXiv  Detail & Related papers  (2022-05-20T20:50:17Z)
• Multivariate feature ranking of gene expression data [62.997667081978825]
  We propose two new multivariate feature ranking methods based on pairwise correlation and pairwise consistency. We statistically prove that the proposed methods outperform the state of the art feature ranking methods Clustering Variation, Chi Squared, Correlation, Information Gain, ReliefF and Significance.
  arXiv  Detail & Related papers  (2021-11-03T17:19:53Z)
• A Comparative Evaluation of Quantification Methods [3.1499058381005227]
  Quantification represents the problem of predicting class distributions in a dataset. A large variety of different algorithms has been proposed in recent years. We compare 24 different methods on overall more than 40 data sets.
  arXiv  Detail & Related papers  (2021-03-04T18:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.