Lightweight Deep Learning for Resource-Constrained Environments: A Survey

• URL: http://arxiv.org/abs/2404.07236v2
• Date: Fri, 12 Apr 2024 09:34:38 GMT
• Title: Lightweight Deep Learning for Resource-Constrained Environments: A Survey
• Authors: Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, Wen-Huang Cheng,
• Abstract summary: Deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. We explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models.
• Score: 30.791667683400444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model's accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.

Related papers

• Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutions [8.313977832587684]
  Object detection (OD) has become vital for numerous computer vision applications, but deploying it on resource-constrained IoT devices presents a significant challenge.<n>TinyML offers a compelling solution by enabling OD on ultra-low-power devices, paving the way for efficient and real-time processing at the edge.<n>This paper provides a detailed analysis of key optimization techniques for deploying OD models on resource-constrained devices.
  arXiv  Detail & Related papers  (2025-08-11T17:28:59Z)
• Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation [75.30238170051291]
  Depth estimation is a fundamental task in 3D computer vision, crucial for applications such as 3D reconstruction, free-viewpoint rendering, robotics, autonomous driving, and AR/VR technologies.<n>Traditional methods relying on hardware sensors like LiDAR are often limited by high costs, low resolution, and environmental sensitivity, limiting their applicability in real-world scenarios.<n>Recent advances in vision-based methods offer a promising alternative, yet they face challenges in generalization and stability due to either the low-capacity model architectures or the reliance on domain-specific and small-scale datasets.
  arXiv  Detail & Related papers  (2025-07-15T17:59:59Z)
• Machine-Learning-Assisted Photonic Device Development: A Multiscale Approach from Theory to Characterization [80.82828320306464]
  Photonic device development (PDD) has achieved remarkable success in designing and implementing new devices for controlling light across various wavelengths, scales, and applications.<n>PDD is an iterative, five-step process that consists of: i.e. deriving device behavior from design parameters, ii. simulating device performance, iv. fabricating the optimal device, and v. measuring device performance.<n>PDD suffers from large optimization landscapes, uncertainties in structural or optical characterization, and difficulties in implementing robust fabrication processes.<n>In this review, we present a comprehensive perspective on these methods to enable machine-learning-assisted PDD
  arXiv  Detail & Related papers  (2025-06-24T23:32:54Z)
• From Tiny Machine Learning to Tiny Deep Learning: A Survey [0.13281177137699654]
  The rapid growth of edge devices has driven the demand for deploying artificial intelligence (AI) at the edge.<n>The emergence of TinyDL marks a paradigm shift toward deploying deep learning models on severely resource-constrained hardware.<n>This survey aims to serve as a foundational resource for researchers and practitioners, offering a holistic view of the ecosystem.
  arXiv  Detail & Related papers  (2025-06-21T05:09:07Z)
• Continual Learning for Generative AI: From LLMs to MLLMs and Beyond [56.29231194002407]
  We present a comprehensive survey of continual learning methods for mainstream generative AI models.<n>We categorize these approaches into three paradigms: architecture-based, regularization-based, and replay-based.<n>We analyze continual learning setups for different generative models, including training objectives, benchmarks, and core backbones.
  arXiv  Detail & Related papers  (2025-06-16T02:27:25Z)
• A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications [7.414857515253022]
  We introduce the basic design of MoE, including gating functions, expert networks, routing mechanisms, training strategies, and system design. We then explore the algorithm design of MoE in important machine learning paradigms such as continual learning, meta-learning, multi-task learning, and reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-10T10:08:55Z)
• On Accelerating Edge AI: Optimizing Resource-Constrained Environments [1.7355861031903428]
  Resource-constrained edge deployments demand AI solutions that balance high performance with stringent compute, memory, and energy limitations. We present a comprehensive overview of the primary strategies for accelerating deep learning models under such constraints.
  arXiv  Detail & Related papers  (2025-01-25T01:37:03Z)
• Lightweight Design and Optimization methods for DCNNs: Progress and Futures [40.96453902709292]
  Deep Convolutional Neural Networks (DCNNs) have demonstrated superior performance in computer vision tasks. High computational costs and large network architectures severely limit the widespread application of DCNNs on resource-constrained hardware platforms. This paper reviews lightweight design strategies for DCNNs and examines recent research progress in both lightweight architectural design and model compression.
  arXiv  Detail & Related papers  (2024-12-22T06:47:01Z)
• Darkit: A User-Friendly Software Toolkit for Spiking Large Language Model [50.37090759139591]
  Large language models (LLMs) have been widely applied in various practical applications, typically comprising billions of parameters. The human brain, employing bio-plausible spiking mechanisms, can accomplish the same tasks while significantly reducing energy consumption. We are releasing a software toolkit named DarwinKit (Darkit) to accelerate the adoption of brain-inspired large language models.
  arXiv  Detail & Related papers  (2024-12-20T07:50:08Z)
• A Survey of Small Language Models [104.80308007044634]
  Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources. We present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques.
  arXiv  Detail & Related papers  (2024-10-25T23:52:28Z)
• On-Device Language Models: A Comprehensive Review [26.759861320845467]
  Review examines the challenges of deploying computationally expensive large language models on resource-constrained devices. Paper investigates on-device language models, their efficient architectures, as well as state-of-the-art compression techniques. Case studies of on-device language models from major mobile manufacturers demonstrate real-world applications and potential benefits.
  arXiv  Detail & Related papers  (2024-08-26T03:33:36Z)
• Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches [64.42735183056062]
  Large language models (LLMs) have transitioned from specialized models to versatile foundation models. LLMs exhibit impressive zero-shot ability, however, require fine-tuning on local datasets and significant resources for deployment.
  arXiv  Detail & Related papers  (2024-08-20T09:42:17Z)
• Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models [0.0]
  An increase in the number of connected devices around the world warrants compressed models that can be easily deployed at the local devices with low compute capacity and power accessibility. We implemented both, quantization and pruning, compression techniques on popular deep learning models used in the image classification, object detection, language models and generative models-based problem statements.
  arXiv  Detail & Related papers  (2024-07-22T14:20:53Z)
• Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
  Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks. Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios. New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
  arXiv  Detail & Related papers  (2023-08-27T03:55:28Z)
• On-device Training: A First Overview on Existing Systems [6.551096686706628]
  Efforts have been made to deploy some models on resource-constrained devices as well. This work targets to summarize and analyze state-of-the-art systems research that allows such on-device model training capabilities.
  arXiv  Detail & Related papers  (2022-12-01T19:22:29Z)
• Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey [53.258091735278875]
  This survey covers studies of design automation techniques for deep learning models targeting edge computing. It offers an overview and comparison of key metrics that are used commonly to quantify the proficiency of models in terms of effectiveness, lightness, and computational costs. The survey proceeds to cover three categories of the state-of-the-art of deep model design automation techniques.
  arXiv  Detail & Related papers  (2022-08-22T12:12:43Z)
• Ultrasound Signal Processing: From Models to Deep Learning [64.56774869055826]
  Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Deep learning based methods, which are optimized in a data-driven fashion, have gained popularity. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning, as well as exploiting domain knowledge.
  arXiv  Detail & Related papers  (2022-04-09T13:04:36Z)
• Knowledge Distillation: A Survey [87.51063304509067]
  Deep neural networks have been successful in both industry and academia, especially for computer vision tasks. It is a challenge to deploy these cumbersome deep models on devices with limited resources. Knowledge distillation effectively learns a small student model from a large teacher model.
  arXiv  Detail & Related papers  (2020-06-09T21:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.