Deep Learning Approach to Anomaly Detection in Enterprise ETL Processes with Autoencoders

• URL: http://arxiv.org/abs/2511.00462v1
• Date: Sat, 01 Nov 2025 09:06:26 GMT
• Title: Deep Learning Approach to Anomaly Detection in Enterprise ETL Processes with Autoencoders
• Authors: Xin Chen, Saili Uday Gadgil, Kangning Gao, Yi Hu, Cong Nie,
• Abstract summary: The study first analyzes multiple types of anomalies in processes, including delays, missing values, duplicate loading, and sudden abnormal changes.<n>In the method design, the encoder-decoder structure compresses high-dimensional inputs into latent representations and reconstructs them, while reconstruction error is used to measure anomaly levels.<n>The results demonstrate that the deep autoencoder-based detection mechanism can effectively capture latent distribution patterns in enterprise-level data streams.
• Score: 10.829077063619247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An anomaly detection method based on deep autoencoders is proposed to address anomalies that often occur in enterprise-level ETL data streams. The study first analyzes multiple types of anomalies in ETL processes, including delays, missing values, duplicate loading, and sudden abnormal changes, and applies data standardization and feature modeling to ensure stable and usable inputs. In the method design, the encoder-decoder structure compresses high-dimensional inputs into latent representations and reconstructs them, while reconstruction error is used to measure anomaly levels. Regularization constraints are introduced in the latent space to enhance feature sparsity and distribution learning, thereby improving robustness in complex data streams. Systematic analyses under different hyperparameter settings, environmental changes, and data characteristics show that the proposed method achieves superior performance in AUC, ACC, Precision, and Recall. The results demonstrate that the deep autoencoder-based detection mechanism can effectively capture latent distribution patterns in enterprise-level ETL data streams and accurately identify diverse anomalies, providing reliable support for enterprise data processing and intelligent analysis.

Related papers

• Real-Time Adaptive Anomaly Detection in Industrial IoT Environments [2.4820910939873806]
  We propose an adaptive method for detecting anomalies in IIoT streaming data utilizing a multi-source prediction model and concept drift adaptation.<n>Our trace-driven evaluations indicate that the proposed method outperforms the state-of-the-art anomaly detection methods by achieving up to an 89.71% accuracy.
  arXiv  Detail & Related papers  (2026-01-06T15:16:45Z)
• Dynamic Anomaly Identification in Accounting Transactions via Multi-Head Self-Attention Networks [1.869032078105649]
  This study addresses the problem of dynamic anomaly detection in accounting transactions.<n>It proposes a real-time detection method based on a Transformer to tackle the challenges of hidden abnormal behaviors.
  arXiv  Detail & Related papers  (2025-11-15T09:18:32Z)
• Counterfactual Explanation for Auto-Encoder Based Time-Series Anomaly Detection [0.3199881502576702]
  Auto-Encoders exhibit inherent opaqueness in their decision-making processes, hindering their practical implementation at scale.<n>In this work, we employ a feature selector to select features and counterfactual explanations to give a context to the model output.<n>Our experimental findings illustrate that our proposed counterfactual approach can offer meaningful and valuable insights into the model decision-making process.
  arXiv  Detail & Related papers  (2025-01-03T19:30:11Z)
• Anomaly detection using Diffusion-based methods [15.049468347670421]
  This paper explores the utility of diffusion-based models for anomaly detection.<n>It focuses on their efficacy in identifying deviations in both compact and high-resolution datasets.
  arXiv  Detail & Related papers  (2024-12-10T14:17:23Z)
• ADT: Agent-based Dynamic Thresholding for Anomaly Detection [4.356615197661274]
  We propose an agent-based dynamic thresholding (ADT) framework based on a deep Q-network. An auto-encoder is utilized in this study to obtain feature representations and produce anomaly scores for complex input data. ADT can adjust thresholds adaptively by utilizing the anomaly scores from the auto-encoder.
  arXiv  Detail & Related papers  (2023-12-03T19:07:30Z)
• A Robust and Explainable Data-Driven Anomaly Detection Approach For Power Electronics [56.86150790999639]
  We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer. The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data. A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
  arXiv  Detail & Related papers  (2022-09-23T06:09:35Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly detection in air transportation [68.8204255655161]
  We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE) It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase. Results show that the DAE achieves better results in both accuracy and speed of detection.
  arXiv  Detail & Related papers  (2021-09-08T14:07:55Z)
• Semantic Perturbations with Normalizing Flows for Improved Generalization [62.998818375912506]
  We show that perturbations in the latent space can be used to define fully unsupervised data augmentations. We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective.
  arXiv  Detail & Related papers  (2021-08-18T03:20:00Z)
• Adaptive Anomaly Detection for Internet of Things in Hierarchical Edge Computing: A Contextual-Bandit Approach [81.5261621619557]
  We propose an adaptive anomaly detection scheme with hierarchical edge computing (HEC) We first construct multiple anomaly detection DNN models with increasing complexity, and associate each of them to a corresponding HEC layer. Then, we design an adaptive model selection scheme that is formulated as a contextual-bandit problem and solved by using a reinforcement learning policy network.
  arXiv  Detail & Related papers  (2021-08-09T08:45:47Z)
• ESAD: End-to-end Deep Semi-supervised Anomaly Detection [85.81138474858197]
  We propose a new objective function that measures the KL-divergence between normal and anomalous data. The proposed method significantly outperforms several state-of-the-arts on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-09T08:16:35Z)
• Real-World Anomaly Detection by using Digital Twin Systems and Weakly-Supervised Learning [3.0100975935933567]
  We present novel weakly-supervised approaches to anomaly detection for industrial settings. The approaches make use of a Digital Twin to generate a training dataset which simulates the normal operation of the machinery. The performance of the proposed methods is compared against various state-of-the-art anomaly detection algorithms on an application to a real-world dataset.
  arXiv  Detail & Related papers  (2020-11-12T10:15:56Z)
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [79.18710225716791]
  Machine learning (ML) systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a distribution different from training data. Existing OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. We propose Neural Architecture Distribution Search (NADS) to identify common building blocks among all uncertainty-aware architectures.
  arXiv  Detail & Related papers  (2020-06-11T17:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.