ENCODE: Encoding NetFlows for Network Anomaly Detection
- URL: http://arxiv.org/abs/2207.03890v2
- Date: Fri, 4 Aug 2023 09:03:40 GMT
- Title: ENCODE: Encoding NetFlows for Network Anomaly Detection
- Authors: Clinton Cao, Annibale Panichella, Sicco Verwer, Agathe Blaise, Filippo
Rebecchi
- Abstract summary: Many works have used machine learning to detect network attacks using NetFlow data.
We propose an encoding algorithm that takes the frequency and context of the feature values into account.
We train several machine learning models for anomaly detection using the data encoded with our algorithm.
- Score: 17.94733537757708
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: NetFlow data is a popular network log format used by many network analysts
and researchers. The advantages of using NetFlow over deep packet inspection
are that it is easier to collect and process, and it is less privacy intrusive.
Many works have used machine learning to detect network attacks using NetFlow
data. The first step for these machine learning pipelines is to pre-process the
data before it is given to the machine learning algorithm. Many approaches
exist to pre-process NetFlow data; however, these simply apply existing methods
to the data, not considering the specific properties of network data. We argue
that for data originating from software systems, such as NetFlow or software
logs, similarities in frequency and contexts of feature values are more
important than similarities in the value itself. In this work, we propose an
encoding algorithm that directly takes the frequency and the context of the
feature values into account when the data is being processed. Different types
of network behaviours can be clustered using this encoding, thus aiding the
process of detecting anomalies within the network. We train several machine
learning models for anomaly detection using the data that has been encoded with
our encoding algorithm. We evaluate the effectiveness of our encoding on a new
dataset that we created for network attacks on Kubernetes clusters and two
well-known public NetFlow datasets. We empirically demonstrate that the machine
learning models benefit from using our encoding for anomaly detection.
Related papers
- DOC-NAD: A Hybrid Deep One-class Classifier for Network Anomaly
Detection [0.0]
Machine Learning approaches have been used to enhance the detection capabilities of Network Intrusion Detection Systems (NIDSs)
Recent work has achieved near-perfect performance by following binary- and multi-class network anomaly detection tasks.
This paper proposes a Deep One-Class (DOC) classifier for network intrusion detection by only training on benign network data samples.
arXiv Detail & Related papers (2022-12-15T00:08:05Z) - Network Security Modelling with Distributional Data [4.133655523622441]
We investigate the detection of botnet command and control (C2) hosts in massive IP traffic using machine learning methods.
We use NetFlow data -- the industry standard for monitoring of IP traffic -- and ML models using two sets of features.
We use quantiles of their IP-level distributions as input features in predictive models to predict whether an IP belongs to known botnet families.
arXiv Detail & Related papers (2022-11-24T05:18:17Z) - From One to Many: A Deep Learning Coincident Gravitational-Wave Search [58.720142291102135]
We construct a two-detector search for gravitational waves from binary black hole mergers using neural networks trained on non-spinning binary black hole data from a single detector.
We find that none of these simple two-detector networks are capable of improving the sensitivity over applying networks individually to the data from the detectors.
arXiv Detail & Related papers (2021-08-24T13:25:02Z) - SignalNet: A Low Resolution Sinusoid Decomposition and Estimation
Network [79.04274563889548]
We propose SignalNet, a neural network architecture that detects the number of sinusoids and estimates their parameters from quantized in-phase and quadrature samples.
We introduce a worst-case learning threshold for comparing the results of our network relative to the underlying data distributions.
In simulation, we find that our algorithm is always able to surpass the threshold for three-bit data but often cannot exceed the threshold for one-bit data.
arXiv Detail & Related papers (2021-06-10T04:21:20Z) - Enabling certification of verification-agnostic networks via
memory-efficient semidefinite programming [97.40955121478716]
We propose a first-order dual SDP algorithm that requires memory only linear in the total number of network activations.
We significantly improve L-inf verified robust accuracy from 1% to 88% and 6% to 40% respectively.
We also demonstrate tight verification of a quadratic stability specification for the decoder of a variational autoencoder.
arXiv Detail & Related papers (2020-10-22T12:32:29Z) - Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks [78.65792427542672]
Dynamic Graph Network (DG-Net) is a complete directed acyclic graph, where the nodes represent convolutional blocks and the edges represent connection paths.
Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network to have more representation ability.
arXiv Detail & Related papers (2020-10-02T16:50:26Z) - Learning the Structure of Auto-Encoding Recommenders [1.9981375888949475]
We introduce structure learning for autoencoder recommenders by taking advantage of the inherent item groups present in the collaborative filtering domain.
Based on this, we propose a method that first learns groups of related items and then uses this information to determine the connectivity structure of an auto-encoding neural network.
The resultant sparse network considerably outperforms the state-of-the-art methods like textscMult-vae/Mult-dae on multiple benchmarked datasets.
arXiv Detail & Related papers (2020-08-18T14:37:40Z) - Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs.
Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-04-06T15:51:00Z) - Fault Handling in Large Water Networks with Online Dictionary Learning [1.933681537640272]
Here we simplify the model by offering a data driven alternative that takes the network topology into account when performing sensor placement.
Online learning is fast and allows tackling large networks as it processes small batches of signals at a time.
The algorithms show good performance when tested on both small and large-scale networks.
arXiv Detail & Related papers (2020-03-18T21:46:14Z) - Automating Botnet Detection with Graph Neural Networks [106.24877728212546]
Botnets are now a major source for many network attacks, such as DDoS attacks and spam.
In this paper, we consider the neural network design challenges of using modern deep learning techniques to learn policies for botnet detection automatically.
arXiv Detail & Related papers (2020-03-13T15:34:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.