A new hope for network model generalization

• URL: http://arxiv.org/abs/2207.05843v1
• Date: Tue, 12 Jul 2022 21:16:38 GMT
• Title: A new hope for network model generalization
• Authors: Alexander Dietm\"uller, Siddhant Ray, Romain Jacob, Laurent Vanbever
• Abstract summary: Generalizing machine learning models for network traffic dynamics tends to be considered a lost cause. An ML architecture called_Transformer_ has enabled previously unimaginable generalization in other domains. We propose a Network Traffic Transformer (NTT) to learn network dynamics from packet traces.
• Score: 66.5377859849467
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generalizing machine learning (ML) models for network traffic dynamics tends to be considered a lost cause. Hence, for every new task, we often resolve to design new models and train them on model-specific datasets collected, whenever possible, in an environment mimicking the model's deployment. This approach essentially gives up on generalization. Yet, an ML architecture called_Transformer_ has enabled previously unimaginable generalization in other domains. Nowadays, one can download a model pre-trained on massive datasets and only fine-tune it for a specific task and context with comparatively little time and data. These fine-tuned models are now state-of-the-art for many benchmarks. We believe this progress could translate to networking and propose a Network Traffic Transformer (NTT), a transformer adapted to learn network dynamics from packet traces. Our initial results are promising: NTT seems able to generalize to new prediction tasks and contexts. This study suggests there is still hope for generalization, though it calls for a lot of future research.

Related papers

• NuwaTS: a Foundation Model Mending Every Incomplete Time Series [24.768755438620666]
  We present textbfNuwaTS, a novel framework that repurposes Pre-trained Language Models for general time series imputation. NuwaTS can be applied to impute missing data across any domain. We show that NuwaTS generalizes to other time series tasks, such as forecasting.
  arXiv  Detail & Related papers  (2024-05-24T07:59:02Z)
• MotherNet: A Foundational Hypernetwork for Tabular Classification [1.9643748953805937]
  We propose a hypernetwork architecture that we call MotherNet, trained on millions of classification tasks. MotherNet replaces training on specific datasets with in-context learning through a single forward pass. The child network generated by MotherNet using in-context learning outperforms neural networks trained using gradient descent on small datasets.
  arXiv  Detail & Related papers  (2023-12-14T01:48:58Z)
• Generalization and Estimation Error Bounds for Model-based Neural Networks [78.88759757988761]
  We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks. We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
  arXiv  Detail & Related papers  (2023-04-19T16:39:44Z)
• $\Delta$-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss [71.46601663956521]
  Models pre-trained on large-scale datasets are often fine-tuned to support newer tasks and datasets that arrive over time. We propose $Delta$-Patching for fine-tuning neural network models in an efficient manner, without the need to store model copies. Our experiments show that $Delta$-Networks outperform earlier model patching work while only requiring a fraction of parameters to be trained.
  arXiv  Detail & Related papers  (2023-03-26T16:39:44Z)
• Online Evolutionary Neural Architecture Search for Multivariate Non-Stationary Time Series Forecasting [72.89994745876086]
  This work presents the Online Neuro-Evolution-based Neural Architecture Search (ONE-NAS) algorithm. ONE-NAS is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks. Results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods.
  arXiv  Detail & Related papers  (2023-02-20T22:25:47Z)
• Pretraining Graph Neural Networks for few-shot Analog Circuit Modeling and Design [68.1682448368636]
  We present a supervised pretraining approach to learn circuit representations that can be adapted to new unseen topologies or unseen prediction tasks. To cope with the variable topological structure of different circuits we describe each circuit as a graph and use graph neural networks (GNNs) to learn node embeddings. We show that pretraining GNNs on prediction of output node voltages can encourage learning representations that can be adapted to new unseen topologies or prediction of new circuit level properties.
  arXiv  Detail & Related papers  (2022-03-29T21:18:47Z)
• Shared and Private VAEs with Generative Replay for Continual Learning [1.90365714903665]
  Continual learning tries to learn new tasks without forgetting previously learned ones. Most of the existing artificial neural network(ANN) models fail, while humans do the same by remembering previous works throughout their life. We show our hybrid model effectively avoids forgetting and achieves state-of-the-art results on visual continual learning benchmarks such as MNIST, Permuted MNIST(QMNIST), CIFAR100, and miniImageNet datasets.
  arXiv  Detail & Related papers  (2021-05-17T06:18:36Z)
• Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network [0.0]
  We show that RNN-transducer models can be effectively adapted to new domains using only small amounts of textual data. We show with multiple ASR evaluation tasks how this method can provide relative gains of 10-45% in target task WER.
  arXiv  Detail & Related papers  (2021-04-22T15:21:41Z)
• Pre-Trained Models for Heterogeneous Information Networks [57.78194356302626]
  We propose a self-supervised pre-training and fine-tuning framework, PF-HIN, to capture the features of a heterogeneous information network. PF-HIN consistently and significantly outperforms state-of-the-art alternatives on each of these tasks, on four datasets.
  arXiv  Detail & Related papers  (2020-07-07T03:36:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.