m4: A Learned Flow-level Network Simulator

• URL: http://arxiv.org/abs/2503.01770v1
• Date: Mon, 03 Mar 2025 17:51:08 GMT
• Title: m4: A Learned Flow-level Network Simulator
• Authors: Chenning Li, Anton A. Zabreyko, Arash Nasr-Esfahany, Kevin Zhao, Prateesh Goyal, Mohammad Alizadeh, Thomas Anderson,
• Abstract summary: Flow-level simulators abstract traffic as continuous flows with dynamically assigned transmission rates.<n>We present m4, an accurate and scalable flow-level simulator that uses machine learning to learn the dynamics of the network of interest.
• Score: 14.802940782072122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow-level simulation is widely used to model large-scale data center networks due to its scalability. Unlike packet-level simulators that model individual packets, flow-level simulators abstract traffic as continuous flows with dynamically assigned transmission rates. While this abstraction enables orders-of-magnitude speedup, it is inaccurate by omitting critical packet-level effects such as queuing, congestion control, and retransmissions. We present m4, an accurate and scalable flow-level simulator that uses machine learning to learn the dynamics of the network of interest. At the core of m4 lies a novel ML architecture that decomposes state transition computations into distinct spatial and temporal components, each represented by a suitable neural network. To efficiently learn the underlying flow-level dynamics, m4 adds dense supervision signals by predicting intermediate network metrics such as remaining flow size and queue length during training. m4 achieves a speedup of up to 104$\times$ over packet-level simulation. Relative to a traditional flow-level simulation, m4 reduces per-flow estimation errors by 45.3% (mean) and 53.0% (p90). For closed-loop applications, m4 accurately predicts network throughput under various congestion control schemes and workloads.

Related papers

• FlowTS: Time Series Generation via Rectified Flow [67.41208519939626]
  FlowTS is an ODE-based model that leverages rectified flow with straight-line transport in probability space.<n>For unconditional setting, FlowTS achieves state-of-the-art performance, with context FID scores of 0.019 and 0.011 on Stock and ETTh datasets.<n>For conditional setting, we have achieved superior performance in solar forecasting.
  arXiv  Detail & Related papers  (2024-11-12T03:03:23Z)
• Reducing Spatial Discretization Error on Coarse CFD Simulations Using an OpenFOAM-Embedded Deep Learning Framework [0.7223509567556214]
  We propose a method for reducing the spatial discretization error of fluid dynamics problems by enhancing the quality of simulations using deep learning. We feed the model with fine-grid data after projecting it to the coarse-grid discretization. We substitute the default differencing scheme for the convection term by a feed-forward neural network that interpolates velocities from cell centers to face values to produce velocities that approximate the down-sampled fine-grid data well.
  arXiv  Detail & Related papers  (2024-05-13T02:59:50Z)
• Energy-Guided Data Sampling for Traffic Prediction with Mini Training Datasets [13.065729535009925]
  We propose an innovative solution that merges Convolutional Neural Networks (CNNs) with Long Short-Term Memory (LSTM) architecture to enhance the prediction of traffic flow dynamics. A key revelation of our research is the feasibility of sampling training data for large traffic systems from simulations conducted on smaller traffic systems.
  arXiv  Detail & Related papers  (2024-03-27T15:57:42Z)
• On the Burstiness of Distributed Machine Learning Traffic [0.0]
  We study the traffic characteristics generated by the training of the ResNet-50 neural network. Our analysis reveals that distributed ML traffic exhibits a very high degree of burstiness on short time scales. We observe that training software orchestrates transmissions in such a way that burst transmissions from different sources within the same application do not result in congestion and packet losses.
  arXiv  Detail & Related papers  (2023-12-30T21:33:59Z)
• Transfer learning-based physics-informed convolutional neural network for simulating flow in porous media with time-varying controls [0.0]
  A physics-informed convolutional neural network is proposed to simulate two phase flow in porous media. finite volume scheme is adopted to discretize flow equations. N Neumann boundary conditions are seamlessly incorporated into the semi-discretized equations.
  arXiv  Detail & Related papers  (2023-10-10T05:29:33Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)
• Continual learning autoencoder training for a particle-in-cell simulation via streaming [52.77024349608834]
  upcoming exascale era will provide a new generation of physics simulations with high resolution. These simulations will have a high resolution, which will impact the training of machine learning models since storing a high amount of simulation data on disk is nearly impossible. This work presents an approach that trains a neural network concurrently to a running simulation without data on a disk.
  arXiv  Detail & Related papers  (2022-11-09T09:55:14Z)
• Liquid Structural State-Space Models [106.74783377913433]
  Liquid-S4 achieves an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4.
  arXiv  Detail & Related papers  (2022-09-26T18:37:13Z)
• Manifold Interpolating Optimal-Transport Flows for Trajectory Inference [64.94020639760026]
  We present a method called Manifold Interpolating Optimal-Transport Flow (MIOFlow) MIOFlow learns, continuous population dynamics from static snapshot samples taken at sporadic timepoints. We evaluate our method on simulated data with bifurcations and merges, as well as scRNA-seq data from embryoid body differentiation, and acute myeloid leukemia treatment.
  arXiv  Detail & Related papers  (2022-06-29T22:19:03Z)
• Emulating Spatio-Temporal Realizations of Three-Dimensional Isotropic Turbulence via Deep Sequence Learning Models [24.025975236316842]
  We use a data-driven approach to model a three-dimensional turbulent flow using cutting-edge Deep Learning techniques. The accuracy of the model is assessed using statistical and physics-based metrics.
  arXiv  Detail & Related papers  (2021-12-07T03:33:39Z)
• On the Tradeoff between Energy, Precision, and Accuracy in Federated Quantized Neural Networks [68.52621234990728]
  Federated learning (FL) over wireless networks requires balancing between accuracy, energy efficiency, and precision. We propose a quantized FL framework that represents data with a finite level of precision in both local training and uplink transmission. Our framework can reduce energy consumption by up to 53% compared to a standard FL model.
  arXiv  Detail & Related papers  (2021-11-15T17:00:03Z)
• Recurrent convolutional neural network for the surrogate modeling of subsurface flow simulation [0.0]
  We propose to combine SegNet with ConvLSTM layers for the surrogate modeling of numerical flow simulation. Results show that the proposed method improves the performance of SegNet based surrogate model remarkably when the output of the simulation is time series data.
  arXiv  Detail & Related papers  (2020-10-08T09:34:48Z)
• ML-LBM: Machine Learning Aided Flow Simulation in Porous Media [0.0]
  Direct simulation of fluid flow in porous media requires significant computational resources to solve within reasonable timeframes. An integrated method combining predictions of fluid flow with direct flow simulation is outlined. Deep Learning techniques based on Convolutional Neural Networks (CNNs) are shown to give an accurate estimate of the steady state velocity fields.
  arXiv  Detail & Related papers  (2020-04-22T01:55:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.