Concept Drift and Covariate Shift Detection Ensemble with Lagged Labels
- URL: http://arxiv.org/abs/2012.04759v3
- Date: Tue, 15 Dec 2020 03:49:59 GMT
- Title: Concept Drift and Covariate Shift Detection Ensemble with Lagged Labels
- Authors: Yiming Xu, Diego Klabjan
- Abstract summary: It is important to detect changes and retrain the model in time.
Current methods have three weaknesses: 1) using only classification error rate as signal, 2) assuming ground truth labels are immediately available after features from samples are received and 3) unable to decide what data to use to retrain the model when change occurs.
We address the first problem by utilizing six different signals to capture a wide range of characteristics of data, and we address the second problem by allowing lag of labels, where labels of corresponding features are received after a lag in time.
For the third problem, our proposed method automatically decides what data to use to retrain
- Score: 22.826118321715455
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In model serving, having one fixed model during the entire often life-long
inference process is usually detrimental to model performance, as data
distribution evolves over time, resulting in lack of reliability of the model
trained on historical data. It is important to detect changes and retrain the
model in time. The existing methods generally have three weaknesses: 1) using
only classification error rate as signal, 2) assuming ground truth labels are
immediately available after features from samples are received and 3) unable to
decide what data to use to retrain the model when change occurs. We address the
first problem by utilizing six different signals to capture a wide range of
characteristics of data, and we address the second problem by allowing lag of
labels, where labels of corresponding features are received after a lag in
time. For the third problem, our proposed method automatically decides what
data to use to retrain based on the signals. Extensive experiments on
structured and unstructured data for different type of data changes establish
that our method consistently outperforms the state-of-the-art methods by a
large margin.
Related papers
- Cyber-Resilient System Identification for Power Grid through Bayesian Integration [49.3054872760439]
Power grids increasingly need real-time situational awareness under the ever-evolving cyberthreat landscape.<n>This work advances system identification that combines snapshot-based method with time-series model via Bayesian Integration.
arXiv Detail & Related papers (2025-10-15T19:32:09Z) - Towards Zero-shot 3D Anomaly Localization [58.62650061201283]
3DzAL is a novel patch-level contrastive learning framework for 3D anomaly detection and localization.
We show that 3DzAL outperforms the state-of-the-art anomaly detection and localization performance.
arXiv Detail & Related papers (2024-12-05T16:25:27Z) - LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised
Time Series Anomaly Detection [49.52429991848581]
We propose a Light and Anti-overfitting Retraining Approach (LARA) for deep variational auto-encoder based time series anomaly detection methods (VAEs)
This work aims to make three novel contributions: 1) the retraining process is formulated as a convex problem and can converge at a fast rate as well as prevent overfitting; 2) designing a ruminate block, which leverages the historical data without the need to store them; and 3) mathematically proving that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones.
arXiv Detail & Related papers (2023-10-09T12:36:16Z) - Robust Outlier Rejection for 3D Registration with Variational Bayes [70.98659381852787]
We develop a novel variational non-local network-based outlier rejection framework for robust alignment.
We propose a voting-based inlier searching strategy to cluster the high-quality hypothetical inliers for transformation estimation.
arXiv Detail & Related papers (2023-04-04T03:48:56Z) - Informative regularization for a multi-layer perceptron RR Lyrae
classifier under data shift [3.303002683812084]
We propose a scalable and easily adaptable approach based on an informative regularization and an ad-hoc training procedure to mitigate the shift problem.
Our method provides a new path to incorporate knowledge from characteristic features into artificial neural networks to manage the underlying data shift problem.
arXiv Detail & Related papers (2023-03-12T02:49:19Z) - Learning to Unlearn: Instance-wise Unlearning for Pre-trained
Classifiers [71.70205894168039]
We consider instance-wise unlearning, of which the goal is to delete information on a set of instances from a pre-trained model.
We propose two methods that reduce forgetting on the remaining data: 1) utilizing adversarial examples to overcome forgetting at the representation-level and 2) leveraging weight importance metrics to pinpoint network parameters guilty of propagating unwanted information.
arXiv Detail & Related papers (2023-01-27T07:53:50Z) - Time-Varying Propensity Score to Bridge the Gap between the Past and Present [104.46387765330142]
We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data.
We demonstrate different ways of implementing it and evaluate it on a variety of problems.
arXiv Detail & Related papers (2022-10-04T07:21:49Z) - Data Feedback Loops: Model-driven Amplification of Dataset Biases [9.773315369593876]
We formalize a system where interactions with one model are recorded as history and scraped as training data in the future.
We analyze its stability over time by tracking changes to a test-time bias statistic.
We find that the degree of bias amplification is closely linked to whether the model's outputs behave like samples from the training distribution.
arXiv Detail & Related papers (2022-09-08T17:35:51Z) - Latent Outlier Exposure for Anomaly Detection with Contaminated Data [31.446666264334528]
Anomaly detection aims at identifying data points that show systematic deviations from the majority of data in an unlabeled dataset.
We propose a strategy for training an anomaly detector in the presence of unlabeled anomalies that is compatible with a broad class of models.
arXiv Detail & Related papers (2022-02-16T14:21:28Z) - Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D
Object Detection [85.11649974840758]
3D object detection networks tend to be biased towards the data they are trained on.
We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors.
arXiv Detail & Related papers (2021-11-30T18:42:42Z) - Semi-supervised Long-tailed Recognition using Alternate Sampling [95.93760490301395]
Main challenges in long-tailed recognition come from the imbalanced data distribution and sample scarcity in its tail classes.
We propose a new recognition setting, namely semi-supervised long-tailed recognition.
We demonstrate significant accuracy improvements over other competitive methods on two datasets.
arXiv Detail & Related papers (2021-05-01T00:43:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.