An Incremental Clustering Method for Anomaly Detection in Flight Data
- URL: http://arxiv.org/abs/2005.09874v4
- Date: Wed, 6 Oct 2021 04:32:18 GMT
- Title: An Incremental Clustering Method for Anomaly Detection in Flight Data
- Authors: Weizun Zhao (1), Lishuai Li (2 and 1), Sameer Alam (3), Yanjun Wang
(4) ((1) Department of Systems Engineering and Engineering Management, City
University of Hong Kong, (2) Air Transport and Operations, Faculty of
Aerospace Engineering, Delft University of Technology, (3) School of
Mechanical & Aerospace Engineering, Nanyang Technological University, (4)
College of Civil Aviation, Nanjing University of Aeronautics and
Astronautics)
- Abstract summary: We propose a novel incremental anomaly detection method based on Gaussian Mixture Model (GMM)
It is a probabilistic clustering model of flight operations that can incrementally update its clusters based on new data.
Preliminary results indicate that the incremental learning scheme is effective in dealing with dynamically growing data in flight data analytics.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Safety is a top priority for civil aviation. New anomaly detection methods,
primarily clustering methods, have been developed to monitor pilot operations
and detect any risks from such flight data. However, all existing anomaly
detection methods are offlline learning - the models are trained once using
historical data and used for all future predictions. In practice, new flight
data are accumulated continuously and analyzed every month at airlines.
Clustering such dynamically growing data is challenging for an offlline method
because it is memory and time intensive to re-train the model every time new
data come in. If the model is not re-trained, false alarms or missed detections
may increase since the model cannot reflect changes in data patterns. To
address this problem, we propose a novel incremental anomaly detection method
based on Gaussian Mixture Model (GMM) to identify common patterns and detect
outliers in flight operations from digital flight data. It is a probabilistic
clustering model of flight operations that can incrementally update its
clusters based on new data rather than to re-cluster all data from scratch. It
trains an initial GMM model based on historical offlline data. Then, it
continuously adapts to new incoming data points via an expectation-maximization
(EM) algorithm. To track changes in flight operation patterns, only model
parameters need to be saved. The proposed method was tested on three sets of
simulation data and two sets of real-world flight data. Compared with the
traditional offline GMM method, the proposed method can generate similar
clustering results with significantly reduced processing time (57 % - 99 % time
reduction in testing sets) and memory usage (91 % - 95 % memory usage reduction
in testing sets). Preliminary results indicate that the incremental learning
scheme is effective in dealing with dynamically growing data in flight data
analytics.
Related papers
- Learning Augmentation Policies from A Model Zoo for Time Series Forecasting [58.66211334969299]
We introduce AutoTSAug, a learnable data augmentation method based on reinforcement learning.
By augmenting the marginal samples with a learnable policy, AutoTSAug substantially improves forecasting performance.
arXiv Detail & Related papers (2024-09-10T07:34:19Z) - LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised
Time Series Anomaly Detection [49.52429991848581]
We propose a Light and Anti-overfitting Retraining Approach (LARA) for deep variational auto-encoder based time series anomaly detection methods (VAEs)
This work aims to make three novel contributions: 1) the retraining process is formulated as a convex problem and can converge at a fast rate as well as prevent overfitting; 2) designing a ruminate block, which leverages the historical data without the need to store them; and 3) mathematically proving that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones.
arXiv Detail & Related papers (2023-10-09T12:36:16Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Monte Carlo EM for Deep Time Series Anomaly Detection [6.312089019297173]
Time series data are often corrupted by outliers or other kinds of anomalies.
Recent approaches to anomaly detection and forecasting assume that the proportion of anomalies in the training data is small enough to ignore.
We present a technique for augmenting existing time series models so that they explicitly account for anomalies in the training data.
arXiv Detail & Related papers (2021-12-29T07:52:36Z) - AirLoop: Lifelong Loop Closure Detection [5.3759730885842725]
AirLoop is a method that leverages techniques from lifelong learning to minimize forgetting when training loop closure detection models incrementally.
We experimentally demonstrate the effectiveness of AirLoop on TartanAir, Nordland, and RobotCar datasets.
arXiv Detail & Related papers (2021-09-18T17:28:47Z) - DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly
detection in air transportation [68.8204255655161]
We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE)
It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase.
Results show that the DAE achieves better results in both accuracy and speed of detection.
arXiv Detail & Related papers (2021-09-08T14:07:55Z) - Unsupervised Model Drift Estimation with Batch Normalization Statistics
for Dataset Shift Detection and Model Selection [0.0]
We propose a novel method of model drift estimation by exploiting statistics of batch normalization layer on unlabeled test data.
We show the effectiveness of our method not only on dataset shift detection but also on model selection when there are multiple candidate models among model zoo or training trajectories in an unsupervised way.
arXiv Detail & Related papers (2021-07-01T03:04:47Z) - Continual Learning for Fake Audio Detection [62.54860236190694]
This paper proposes detecting fake without forgetting, a continual-learning-based method, to make the model learn new spoofing attacks incrementally.
Experiments are conducted on the ASVspoof 2019 dataset.
arXiv Detail & Related papers (2021-04-15T07:57:05Z) - Data from Model: Extracting Data from Non-robust and Robust Models [83.60161052867534]
This work explores the reverse process of generating data from a model, attempting to reveal the relationship between the data and the model.
We repeat the process of Data to Model (DtM) and Data from Model (DfM) in sequence and explore the loss of feature mapping information.
Our results show that the accuracy drop is limited even after multiple sequences of DtM and DfM, especially for robust models.
arXiv Detail & Related papers (2020-07-13T05:27:48Z) - Categorical anomaly detection in heterogeneous data using minimum
description length clustering [3.871148938060281]
We propose a meta-algorithm for enhancing any MDL-based anomaly detection model to deal with heterogeneous data.
Our experimental results show that using a discrete mixture model provides competitive performance relative to two previous anomaly detection algorithms.
arXiv Detail & Related papers (2020-06-14T14:48:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.