Are Deep Sequence Classifiers Good at Non-Trivial Generalization?

• URL: http://arxiv.org/abs/2210.13082v1
• Date: Mon, 24 Oct 2022 10:01:06 GMT
• Title: Are Deep Sequence Classifiers Good at Non-Trivial Generalization?
• Authors: Francesco Cazzaro, Ariadna Quattoni, Xavier Carreras
• Abstract summary: We study binary sequence classification problems and we look at model calibration from a different perspective. We focus on sparse sequence classification, that is problems in which the target class is rare and compare three deep learning sequence classification models. Our results suggest that in this binary setting the deep-learning models are indeed able to learn the underlying class distribution in a non-trivial manner.
• Score: 4.941630596191806
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent advances in deep learning models for sequence classification have greatly improved their classification accuracy, specially when large training sets are available. However, several works have suggested that under some settings the predictions made by these models are poorly calibrated. In this work we study binary sequence classification problems and we look at model calibration from a different perspective by asking the question: Are deep learning models capable of learning the underlying target class distribution? We focus on sparse sequence classification, that is problems in which the target class is rare and compare three deep learning sequence classification models. We develop an evaluation that measures how well a classifier is learning the target class distribution. In addition, our evaluation disentangles good performance achieved by mere compression of the training sequences versus performance achieved by proper model generalization. Our results suggest that in this binary setting the deep-learning models are indeed able to learn the underlying class distribution in a non-trivial manner, i.e. by proper generalization beyond data compression.

Related papers

• Deep Imbalanced Regression via Hierarchical Classification Adjustment [50.19438850112964]
  Regression tasks in computer vision are often formulated into classification by quantizing the target space into classes. The majority of training samples lie in a head range of target values, while a minority of samples span a usually larger tail range. We propose to construct hierarchical classifiers for solving imbalanced regression tasks. Our novel hierarchical classification adjustment (HCA) for imbalanced regression shows superior results on three diverse tasks.
  arXiv  Detail & Related papers  (2023-10-26T04:54:39Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• Class-Incremental Learning with Strong Pre-trained Models [97.84755144148535]
  Class-incremental learning (CIL) has been widely studied under the setting of starting from a small number of classes (base classes) We explore an understudied real-world setting of CIL that starts with a strong model pre-trained on a large number of base classes. Our proposed method is robust and generalizes to all analyzed CIL settings.
  arXiv  Detail & Related papers  (2022-04-07T17:58:07Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• Class-Incremental Learning with Generative Classifiers [6.570917734205559]
  We propose a new strategy for class-incremental learning: generative classification. Our proposal is to learn the joint distribution p(x,y), factorized as p(x|y)p(y), and to perform classification using Bayes' rule. As a proof-of-principle, here we implement this strategy by training a variational autoencoder for each class to be learned.
  arXiv  Detail & Related papers  (2021-04-20T16:26:14Z)
• One vs Previous and Similar Classes Learning -- A Comparative Study [2.208242292882514]
  This work proposes three learning paradigms which allow trained models to be updated without the need of retraining from scratch. Results show that the proposed paradigms are faster than the baseline at updating, with two of them being faster at training from scratch as well, especially on larger datasets.
  arXiv  Detail & Related papers  (2021-01-05T00:28:38Z)
• Predicting Classification Accuracy When Adding New Unobserved Classes [8.325327265120283]
  We study how a classifier's performance can be used to extrapolate its expected accuracy on a larger, unobserved set of classes. We formulate a robust neural-network-based algorithm, "CleaneX", which learns to estimate the accuracy of such classifiers on arbitrarily large sets of classes.
  arXiv  Detail & Related papers  (2020-10-28T14:37:25Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Initial Classifier Weights Replay for Memoryless Class Incremental Learning [11.230170401360633]
  Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. We propose a different approach based on a vanilla fine tuning backbone. We conduct a thorough evaluation with four public datasets in a memoryless incremental learning setting.
  arXiv  Detail & Related papers  (2020-08-31T16:18:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.