Building for Tomorrow: Assessing the Temporal Persistence of Text Classifiers

• URL: http://arxiv.org/abs/2205.05435v1
• Date: Wed, 11 May 2022 12:21:14 GMT
• Title: Building for Tomorrow: Assessing the Temporal Persistence of Text Classifiers
• Authors: Rabab Alkhalifa, Elena Kochkina, Arkaitz Zubiaga
• Abstract summary: Performance of text classification models can drop over time when new data to be classified is more distant in time from the data used for training. This raises important research questions on the design of text classification models intended to persist over time. We perform longitudinal classification experiments on three datasets spanning between 6 and 19 years.
• Score: 18.367109894193486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Performance of text classification models can drop over time when new data to be classified is more distant in time from the data used for training, due to naturally occurring changes in the data, such as vocabulary change. A solution to this is to continually label new data to retrain the model, which is, however, often unaffordable to be performed regularly due to its associated cost. This raises important research questions on the design of text classification models that are intended to persist over time: do all embedding models and classification algorithms exhibit similar performance drops over time and is the performance drop more prominent in some tasks or datasets than others? With the aim of answering these research questions, we perform longitudinal classification experiments on three datasets spanning between 6 and 19 years. Findings from these experiments inform the design of text classification models with the aim of preserving performance over time, discussing the extent to which one can rely on classification models trained from temporally distant training data, as well as how the characteristics of the dataset impact this.

Related papers

• VIBE: Topic-Driven Temporal Adaptation for Twitter Classification [9.476760540618903]
  We study temporal adaptation, where models trained on past data are tested in the future. Our model, with only 3% of data, significantly outperforms previous state-of-the-art continued-pretraining methods.
  arXiv  Detail & Related papers  (2023-10-16T08:53:57Z)
• Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain [54.67888148566323]
  We introduce three large-scale time series forecasting datasets from the cloud operations domain. We show it is a strong zero-shot baseline and benefits from further scaling, both in model and dataset size. Accompanying these datasets and results is a suite of comprehensive benchmark results comparing classical and deep learning baselines to our pre-trained method.
  arXiv  Detail & Related papers  (2023-10-08T08:09:51Z)
• Exploring Data Redundancy in Real-world Image Classification through Data Selection [20.389636181891515]
  Deep learning models often require large amounts of data for training, leading to increased costs. We present two data valuation metrics based on Synaptic Intelligence and gradient norms, respectively, to study redundancy in real-world image data. Online and offline data selection algorithms are then proposed via clustering and grouping based on the examined data values.
  arXiv  Detail & Related papers  (2023-06-25T03:31:05Z)
• Stubborn Lexical Bias in Data and Models [50.79738900885665]
  We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data. We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
  arXiv  Detail & Related papers  (2023-06-03T20:12:27Z)
• Self-supervised Activity Representation Learning with Incremental Data: An Empirical Study [7.782045150068569]
  This research examines the impact of using a self-supervised representation learning model for time series classification tasks. We analyzed the effect of varying the size, distribution, and source of the unlabeled data on the final classification performance across four public datasets.
  arXiv  Detail & Related papers  (2023-05-01T01:39:55Z)
• Time Series Contrastive Learning with Information-Aware Augmentations [57.45139904366001]
  A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. We propose a new contrastive learning approach with information-aware augmentations, InfoTS, that adaptively selects optimal augmentations for time series representation learning.
  arXiv  Detail & Related papers  (2023-03-21T15:02:50Z)
• Time-Varying Propensity Score to Bridge the Gap between the Past and Present [104.46387765330142]
  We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data. We demonstrate different ways of implementing it and evaluate it on a variety of problems.
  arXiv  Detail & Related papers  (2022-10-04T07:21:49Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Opinions are Made to be Changed: Temporally Adaptive Stance Classification [9.061088449712859]
  We introduce two novel large-scale, longitudinal stance datasets. We evaluate the performance persistence of stance classifiers over time and demonstrate how it decays as the temporal gap between training and testing data increases. We propose and compare several approaches to embedding adaptation and find that the Incremental Temporal Alignment (ITA) model leads to the best results in reducing performance drop over time.
  arXiv  Detail & Related papers  (2021-08-27T19:47:31Z)
• Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation [101.26235068460551]
  Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. Models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. We introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner.
  arXiv  Detail & Related papers  (2020-10-24T08:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.