Running Neural Networks on the NIC

• URL: http://arxiv.org/abs/2009.02353v1
• Date: Fri, 4 Sep 2020 18:35:58 GMT
• Title: Running Neural Networks on the NIC
• Abstract summary: We show that commodity programmable NICs can run neural network inference tasks with low overhead. N3IC can provide 100x lower processing latency, with 1.5x increase in throughput.
• Score: 7.645750193878735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we show that the data plane of commodity programmable (Network Interface Cards) NICs can run neural network inference tasks required by packet monitoring applications, with low overhead. This is particularly important as the data transfer costs to the host system and dedicated machine learning accelerators, e.g., GPUs, can be more expensive than the processing task itself. We design and implement our system -- N3IC -- on two different NICs and we show that it can greatly benefit three different network monitoring use cases that require machine learning inference as first-class-primitive. N3IC can perform inference for millions of network flows per second, while forwarding traffic at 40Gb/s. Compared to an equivalent solution implemented on a general purpose CPU, N3IC can provide 100x lower processing latency, with 1.5x increase in throughput.

Related papers

• LGNNIC: Acceleration of Large-Scale GNN Training using SmartNICs [0.1633272850273525]
  Graph Neural Networks (GNNs) are widely used across domains such as natural sciences, social network analysis, chip design, and recommendation systems.<n>A promising approach is to distribute the graph across multiple remote memory nodes, though this introduces a major bottleneck: inter-node network congestion during training.<n>We propose LGNNIC, a novel inter-node system architecture that leverages SmartNICs co-located with remote memory nodes.
  arXiv  Detail & Related papers  (2026-08-07T19:55:45Z)
• SNAP-V: A RISC-V SoC with Configurable Neuromorphic Acceleration for Small-Scale Spiking Neural Networks [0.0]
  Spiking Neural Networks (SNNs) have gained significant attention in edge computing due to their low power consumption and computational efficiency.<n>This work presents SNAP-V, a RISC-V-based neuromorphic system for small-scale SNN inference.<n>Results show that the proposed solution enables accurate, energy-efficient SNN inference suitable for real-time edge applications.
  arXiv  Detail & Related papers  (2026-03-12T13:52:44Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Embedded Graph Convolutional Networks for Real-Time Event Data Processing on SoC FPGAs [0.815557531820863]
  Event cameras find significant relevance for their integration into embedded real-time systems. One effective approach to ensure the necessary throughput and latency for event processing systems is through the utilisation of graph convolutional networks (GCNs) We introduce a series of hardware-aware optimisations tailored for PointNet++, a GCN architecture designed for point cloud processing.
  arXiv  Detail & Related papers  (2024-06-11T14:47:36Z)
• FPGA-based AI Smart NICs for Scalable Distributed AI Training Systems [62.20308752994373]
  We propose a new smart network interface card (NIC) for distributed AI training systems using field-programmable gate arrays (FPGAs) Our proposed FPGA-based AI smart NIC enhances overall training performance by 1.6x at 6 nodes, with an estimated 2.5x performance improvement at 32 nodes, compared to the baseline system using conventional NICs.
  arXiv  Detail & Related papers  (2022-04-22T21:57:00Z)
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining [58.10436813430554]
  Mini-batch training of graph neural networks (GNNs) requires a lot of computation and data movement. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised.
  arXiv  Detail & Related papers  (2021-10-16T02:41:35Z)
• FLASH: Fast Neural Architecture Search with Hardware Optimization [7.263481020106725]
  Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs) This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform.
  arXiv  Detail & Related papers  (2021-08-01T23:46:48Z)
• Improving the Performance of a NoC-based CNN Accelerator with Gather Support [6.824747267214373]
  Deep learning technology drives the need for an efficient parallel computing architecture for CNNs. The CNN workload introduces many-to-one traffic in addition to one-to-one and one-to-many traffic. We propose to use the gather packet on mesh-based NoCs employing output stationary systolic array in support of many-to-one traffic.
  arXiv  Detail & Related papers  (2021-08-01T23:33:40Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)
• Toward Accurate Platform-Aware Performance Modeling for Deep Neural Networks [0.17499351967216337]
  We provide a machine learning-based method, PerfNetV2, which improves the accuracy of our previous work for modeling the neural network performance on a variety of GPU accelerators. Given an application, the proposed method can be used to predict the inference time and training time of the convolutional neural networks used in the application. Our case studies show that PerfNetV2 yields a mean absolute percentage error within 13.1% on LeNet, AlexNet, and VGG16 on NVIDIA GTX-1080Ti, while the error rate on a previous work published in ICBD 2018 could be as large as 200%.
  arXiv  Detail & Related papers  (2020-12-01T01:42:23Z)
• LogicNets: Co-Designed Neural Networks and Circuits for Extreme-Throughput Applications [6.9276012494882835]
  We present a novel method for designing neural network topologies that directly map to a highly efficient FPGA implementation. We show that the combination of sparsity and low-bit activation quantization results in high-speed circuits with small logic depth and low LUT cost.
  arXiv  Detail & Related papers  (2020-04-06T22:15:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.