XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification

• URL: http://arxiv.org/abs/2310.14957v1
• Date: Mon, 23 Oct 2023 14:00:02 GMT
• Title: XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification
• Authors: Jacqueline H\"ollig, Steffen Thoma, Florian Grimm
• Abstract summary: This paper proposes XTSC-Bench, a benchmarking tool for evaluating TSC explainability methods. We analyze 3 perturbation-, 6 gradient- and 2 example-based explanation methods to TSC showing that improvements in the explainers' robustness and reliability are necessary.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Despite the growing body of work on explainable machine learning in time series classification (TSC), it remains unclear how to evaluate different explainability methods. Resorting to qualitative assessment and user studies to evaluate explainers for TSC is difficult since humans have difficulties understanding the underlying information contained in time series data. Therefore, a systematic review and quantitative comparison of explanation methods to confirm their correctness becomes crucial. While steps to standardized evaluations were taken for tabular, image, and textual data, benchmarking explainability methods on time series is challenging due to a) traditional metrics not being directly applicable, b) implementation and adaption of traditional metrics for time series in the literature vary, and c) varying baseline implementations. This paper proposes XTSC-Bench, a benchmarking tool providing standardized datasets, models, and metrics for evaluating explanation methods on TSC. We analyze 3 perturbation-, 6 gradient- and 2 example-based explanation methods to TSC showing that improvements in the explainers' robustness and reliability are necessary, especially for multivariate data.

Related papers

• StaICC: Standardized Evaluation for Classification Task in In-context Learning [3.0531121420837226]
  This paper proposes a standardized and easy-to-use evaluation toolkit (StaICC) for in-context classification. For the normal classification task, we provide StaICC-Normal, selecting 10 widely used datasets, and generating prompts with a fixed form. We also provide a sub-benchmark StaICC-Diag for diagnosing ICL from several aspects, aiming for a more robust inference processing.
  arXiv  Detail & Related papers  (2025-01-27T00:05:12Z)
• Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification [4.588028371034407]
  We focus on analyzing InterpretTime, a recent evaluation methodology for attribution methods applied to MTSC. We showcase some significant weaknesses of the original methodology and propose ideas to improve its accuracy and efficiency. We find that perturbation-based methods such as SHAP and Feature Ablation work well across a set of datasets.
  arXiv  Detail & Related papers  (2024-06-18T11:18:46Z)
• Shapelet-based Model-agnostic Counterfactual Local Explanations for Time Series Classification [5.866975269666861]
  We propose a model-agnostic instance-based post-hoc explainability method for time series classification. The proposed algorithm, namely Time-CF, leverages shapelets and TimeGAN to provide counterfactual explanations for arbitrary time series classifiers.
  arXiv  Detail & Related papers  (2024-02-02T11:57:53Z)
• Convolutional and Deep Learning based techniques for Time Series Ordinal Classification [7.047582157120573]
  Time Series Ordinal Classification (TSOC) is the field covering this gap, yet unexplored in the literature. This paper presents a first benchmarking of TSOC methodologies, exploiting the ordering of the target labels to boost the performance of current TSC state-of-the-art.
  arXiv  Detail & Related papers  (2023-06-16T11:57:11Z)
• Robust Explainer Recommendation for Time Series Classification [4.817429789586127]
  Time series classification is a task common in domains such as human activity recognition, sports analytics and general sensing. Recently, a great variety of techniques have been proposed and adapted for time series to provide explanation in the form of saliency maps. This paper provides a novel framework to quantitatively evaluate and rank explanation methods for time series classification.
  arXiv  Detail & Related papers  (2023-06-08T18:49:23Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• Early Time-Series Classification Algorithms: An Empirical Comparison [59.82930053437851]
  Early Time-Series Classification (ETSC) is the task of predicting the class of incoming time-series by observing as few measurements as possible. We evaluate six existing ETSC algorithms on publicly available data, as well as on two newly introduced datasets.
  arXiv  Detail & Related papers  (2022-03-03T10:43:56Z)
• Re-Assessing the "Classify and Count" Quantification Method [88.60021378715636]
  "Classify and Count" (CC) is often a biased estimator. Previous works have failed to use properly optimised versions of CC. We argue that, while still inferior to some cutting-edge methods, they deliver near-state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2020-11-04T21:47:39Z)
• Benchmarking Multivariate Time Series Classification Algorithms [69.12151492736524]
  Time Series Classification (TSC) involved building predictive models for a discrete target variable from ordered, real valued, attributes. Over recent years, a new set of TSC algorithms have been developed which have made significant improvement over the previous state of the art. We review recently proposed bespoke MTSC algorithms based on deep learning, shapelets and bag of words approaches.
  arXiv  Detail & Related papers  (2020-07-26T15:56:40Z)
• AdaS: Adaptive Scheduling of Stochastic Gradients [50.80697760166045]
  We introduce the notions of textit"knowledge gain" and textit"mapping condition" and propose a new algorithm called Adaptive Scheduling (AdaS) Experimentation reveals that, using the derived metrics, AdaS exhibits: (a) faster convergence and superior generalization over existing adaptive learning methods; and (b) lack of dependence on a validation set to determine when to stop training.
  arXiv  Detail & Related papers  (2020-06-11T16:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.