Open World Autoencoding Drift Detection with Novel Class Recognition in Tabular Non-stationary Data Streams

• URL: http://arxiv.org/abs/2605.29834v1
• Date: Thu, 28 May 2026 12:12:30 GMT
• Title: Open World Autoencoding Drift Detection with Novel Class Recognition in Tabular Non-stationary Data Streams
• Abstract summary: This work proposes an unsupervised concept drift detection method.<n>It identifies shifts in known class distributions based on the reconstruction errors of an autoencoder.<n>It also enables the recognition of novel class samples through density estimation of a proxy representation.
• Score: 0.2538209532048867
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Data stream processing has become a landmark in modern machine learning applications, with concept drifts and novel class appearances posing the primary challenges faced by sophisticated recognition methods. This work proposes an unsupervised concept drift detection method that identifies shifts in known class distributions based on the reconstruction errors of an autoencoder, while also enabling the recognition of novel class samples through density estimation of a proxy representation of samples. Using mirrored autoencoders allows for independent incremental adaptation to changing problem distributions for the two considered tasks, resulting in continuous adjustment to evolving concepts and reliable recognition of unknown samples. Conducted experiments used a diverse set of synthetic tabular data streams, where both concept drifts and the emergence of novelties were observed. The results show that the proposed approach is competitive with current state-of-the-art unsupervised drift detectors and novelty classifiers.

Related papers

• Synthetic Non-stationary Data Streams for Recognition of the Unknown [0.0]
  This article presents a strategy for synthetic data stream generation in which concept drifts and the emergence of new classes occur.<n>It shows how unsupervised drift detectors address the task of detecting novelty and concept drifts.
  arXiv  Detail & Related papers  (2025-05-19T21:44:32Z)
• CONCLAD: COntinuous Novel CLAss Detector [5.857367484128867]
  We present a comprehensive solution to the problem of continual novel class detection in post-deployment data.<n>We employ an iterative uncertainty estimation algorithm to differentiate between known and novel class(es) samples.<n>We will release our code upon acceptance.
  arXiv  Detail & Related papers  (2024-12-13T01:41:28Z)
• A Neighbor-Searching Discrepancy-based Drift Detection Scheme for Learning Evolving Data [40.00357483768265]
  This work presents a novel real concept drift detection method based on Neighbor-Searching Discrepancy. The proposed method is able to detect real concept drift with high accuracy while ignoring virtual drift. It can also indicate the direction of the classification boundary change by identifying the invasion or retreat of a certain class.
  arXiv  Detail & Related papers  (2024-05-23T04:03:36Z)
• Combating Concept Drift with Explanatory Detection and Adaptation for Android Malware Classification [17.399454244765842]
  DREAM is a novel system that improves drift detection and establishes an explanatory adaptation process.<n>Our evaluation shows that DREAM effectively improves the drift detection accuracy and reduces the expert analysis effort in adaptation.
  arXiv  Detail & Related papers  (2024-05-07T07:55:45Z)
• Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation [49.827306773992376]
  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions. Our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-12-19T15:34:52Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• Novelty Detection via Contrastive Learning with Negative Data Augmentation [34.39521195691397]
  We introduce a novel generative network framework for novelty detection. Our model has significant superiority over cutting-edge novelty detectors. Our model is more stable for training in a non-adversarial manner, compared to other adversarial based novelty detection methods.
  arXiv  Detail & Related papers  (2021-06-18T07:26:15Z)
• Semi-supervised Long-tailed Recognition using Alternate Sampling [95.93760490301395]
  Main challenges in long-tailed recognition come from the imbalanced data distribution and sample scarcity in its tail classes. We propose a new recognition setting, namely semi-supervised long-tailed recognition. We demonstrate significant accuracy improvements over other competitive methods on two datasets.
  arXiv  Detail & Related papers  (2021-05-01T00:43:38Z)
• Conditional Variational Capsule Network for Open Set Recognition [64.18600886936557]
  In open set recognition, a classifier has to detect unknown classes that are not known at training time. Recently proposed Capsule Networks have shown to outperform alternatives in many fields, particularly in image recognition. In our proposal, during training, capsules features of the same known class are encouraged to match a pre-defined gaussian, one for each class.
  arXiv  Detail & Related papers  (2021-04-19T09:39:30Z)
• Open Set Recognition with Conditional Probabilistic Generative Models [51.40872765917125]
  We propose Conditional Probabilistic Generative Models (CPGM) for open set recognition. CPGM can detect unknown samples but also classify known classes by forcing different latent features to approximate conditional Gaussian distributions. Experiment results on multiple benchmark datasets reveal that the proposed method significantly outperforms the baselines.
  arXiv  Detail & Related papers  (2020-08-12T06:23:49Z)
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders [51.691585766702744]
  We propose a variant of Adversarial Autoencoder which uses a mirrored Wasserstein loss in the discriminator to enforce better semantic-level reconstruction. We put forward an alternative measure of anomaly score to replace the reconstruction-based metric. Our method outperforms the current state-of-the-art methods for anomaly detection on several OOD detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-24T08:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.