Early-Stage Anomaly Detection: A Study of Model Performance on Complete vs. Partial Flows

• URL: http://arxiv.org/abs/2407.02856v2
• Date: Wed, 19 Feb 2025 15:25:31 GMT
• Title: Early-Stage Anomaly Detection: A Study of Model Performance on Complete vs. Partial Flows
• Authors: Adrian Pekar, Richard Jozsa,
• Abstract summary: This study investigates the efficacy of machine learning models in network anomaly detection through the critical lens of partial versus complete flow information. We demonstrate a significant performance difference when models trained on complete flows are tested against partial flows. The study reveals that a minimum of 7 packets in the test set is required for maintaining reliable detection rates.
• Score: 0.0
• License:
• Abstract: This study investigates the efficacy of machine learning models in network anomaly detection through the critical lens of partial versus complete flow information. We systematically evaluate how models perform under varying training and testing conditions, quantifying the performance impact when dealing with incomplete data typical in real-time environments. Our findings demonstrate a significant performance difference, with precision and recall dropping by up to 30% under certain conditions when models trained on complete flows are tested against partial flows. Conversely, models trained and tested on consistently complete or partial datasets maintain robustness. The study reveals that a minimum of 7 packets in the test set is required for maintaining reliable detection rates, providing valuable insights for real-time detection strategies. These results offer important guidance for deploying machine learning models in operational network security environments.

Related papers

• Complementary Learning for Real-World Model Failure Detection [15.779651238128562]
  We introduce complementary learning, where we use learned characteristics from different training paradigms to detect model errors. We demonstrate our approach by learning semantic and predictive motion labels in point clouds in a supervised and self-supervised manner. We perform a large-scale qualitative analysis and present LidarCODA, the first dataset with labeled anomalies in lidar point clouds.
  arXiv  Detail & Related papers  (2024-07-19T13:36:35Z)
• The Importance of Model Inspection for Better Understanding Performance Characteristics of Graph Neural Networks [15.569758991934934]
  We investigate the effect of modelling choices on the feature learning characteristics of graph neural networks applied to a brain shape classification task. We find substantial differences in the feature embeddings at different layers of the models.
  arXiv  Detail & Related papers  (2024-05-02T13:26:18Z)
• Monitoring Machine Learning Models: Online Detection of Relevant Deviations [0.0]
  Machine learning models can degrade over time due to changes in data distribution or other factors. We propose a sequential monitoring scheme to detect relevant changes. Our research contributes a practical solution for distinguishing between minor fluctuations and meaningful degradations.
  arXiv  Detail & Related papers  (2023-09-26T18:46:37Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• Provable Robustness for Streaming Models with a Sliding Window [51.85182389861261]
  In deep learning applications such as online content recommendation and stock market analysis, models use historical data to make predictions. We derive robustness certificates for models that use a fixed-size sliding window over the input stream. Our guarantees hold for the average model performance across the entire stream and are independent of stream size, making them suitable for large data streams.
  arXiv  Detail & Related papers  (2023-03-28T21:02:35Z)
• DELTA: degradation-free fully test-time adaptation [59.74287982885375]
  We find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes.
  arXiv  Detail & Related papers  (2023-01-30T15:54:00Z)
• A monitoring framework for deployed machine learning models with supply chain examples [2.904613270228912]
  We describe a framework for monitoring machine learning models; and, (2) its implementation for a big data supply chain application. We use our implementation to study drift in model features, predictions, and performance on three real data sets.
  arXiv  Detail & Related papers  (2022-11-11T14:31:38Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning [25.85044477227461]
  Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" We find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
  arXiv  Detail & Related papers  (2021-06-30T06:21:42Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.