Generative Optimization Networks for Memory Efficient Data Generation

• URL: http://arxiv.org/abs/2110.02912v2
• Date: Thu, 7 Oct 2021 10:00:51 GMT
• Title: Generative Optimization Networks for Memory Efficient Data Generation
• Authors: Shreshth Tuli, Shikhar Tuli, Giuliano Casale and Nicholas R. Jennings
• Abstract summary: We propose a novel framework called generative optimization networks (GON) that is similar to GANs, but does not use a generator. GONs use a single discriminator network and run optimization in the input space to generate new data samples, achieving an effective compromise between training time and memory consumption. We show that our framework gives up to 32% higher detection F1 scores and 58% lower memory consumption, with only 5% higher training overheads compared to the state-of-the-art.
• Score: 11.452816167207937
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In standard generative deep learning models, such as autoencoders or GANs, the size of the parameter set is proportional to the complexity of the generated data distribution. A significant challenge is to deploy resource-hungry deep learning models in devices with limited memory to prevent system upgrade costs. To combat this, we propose a novel framework called generative optimization networks (GON) that is similar to GANs, but does not use a generator, significantly reducing its memory footprint. GONs use a single discriminator network and run optimization in the input space to generate new data samples, achieving an effective compromise between training time and memory consumption. GONs are most suited for data generation problems in limited memory settings. Here we illustrate their use for the problem of anomaly detection in memory-constrained edge devices arising from attacks or intrusion events. Specifically, we use a GON to calculate a reconstruction-based anomaly score for input time-series windows. Experiments on a Raspberry-Pi testbed with two existing and a new suite of datasets show that our framework gives up to 32% higher detection F1 scores and 58% lower memory consumption, with only 5% higher training overheads compared to the state-of-the-art.

Related papers

• Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT Devices [9.928745904761358]
  Edge intelligence enables resource-demanding Deep Neural Network (DNN) inference without transferring original data. For privacy-sensitive applications, deploying models in hardware-isolated trusted execution environments (TEEs) becomes essential. We present a novel approach for advanced model deployment in TrustZone that ensures comprehensive privacy preservation during model inference.
  arXiv  Detail & Related papers  (2024-03-19T09:22:50Z)
• Topology-aware Embedding Memory for Continual Learning on Expanding Networks [63.35819388164267]
  We present a framework to tackle the memory explosion problem using memory replay techniques. PDGNNs with Topology-aware Embedding Memory (TEM) significantly outperform state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-01-24T03:03:17Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• TinyAD: Memory-efficient anomaly detection for time series data in Industrial IoT [43.207210990362825]
  We propose a novel framework named Tiny Anomaly Detection (TinyAD) to efficiently facilitate onboard inference of CNNs for real-time anomaly detection. To reduce the peak memory consumption of CNNs, we explore two complementary strategies, in-place, and patch-by-patch memory rescheduling. Our framework can reduce peak memory consumption by 2-5x with negligible overhead.
  arXiv  Detail & Related papers  (2023-03-07T02:56:15Z)
• DRAGON: Decentralized Fault Tolerance in Edge Federations [13.864161788250856]
  We propose a novel memory-efficient deep learning based model, namely generative optimization networks (GON) GONs use a single network to both discriminate input and generate samples, significantly reducing their memory footprint. We propose a decentralized fault-tolerance method called DRAGON that runs simulations to quickly predict and optimize the performance of the edge federation.
  arXiv  Detail & Related papers  (2022-08-16T10:40:28Z)
• On-Device Training Under 256KB Memory [62.95579393237751]
  We propose an algorithm-system co-design framework to make on-device training possible with only 256KB of memory. Our framework is the first solution to enable tiny on-device training of convolutional neural networks under 256KB and 1MB Flash.
  arXiv  Detail & Related papers  (2022-06-30T17:59:08Z)
• Memory Replay with Data Compression for Continual Learning [80.95444077825852]
  We propose memory replay with data compression to reduce the storage cost of old training samples. We extensively validate this across several benchmarks of class-incremental learning and in a realistic scenario of object detection for autonomous driving.
  arXiv  Detail & Related papers  (2022-02-14T10:26:23Z)
• SreaMRAK a Streaming Multi-Resolution Adaptive Kernel Algorithm [60.61943386819384]
  Existing implementations of KRR require that all the data is stored in the main memory. We propose StreaMRAK - a streaming version of KRR. We present a showcase study on two synthetic problems and the prediction of the trajectory of a double pendulum.
  arXiv  Detail & Related papers  (2021-08-23T21:03:09Z)
• MAFAT: Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference [1.7894377200944507]
  Machine learning networks can easily exceed available memory, increasing latency due to excessive OS swapping. We propose a memory usage predictor coupled with a search algorithm to provide optimized fusing and tiling configurations. Results show that our approach can run in less than half the memory, and with a speedup of up to 2.78 under severe memory constraints.
  arXiv  Detail & Related papers  (2021-07-14T19:45:49Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.