EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems

• URL: http://arxiv.org/abs/2607.09063v1
• Date: Fri, 10 Jul 2026 03:16:10 GMT
• Title: EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
• Abstract summary: Edge devices are increasingly utilized for deploying deep learning applications on embedded systems.<n>This letter presents a novel and efficient framework, named EvoLP, to accurately predict the inference latency of models on edge devices.
• Score: 12.681921492194986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Edge devices are increasingly utilized for deploying deep learning applications on embedded systems. The real-time nature of many applications and the limited resources of edge devices necessitate latency-targeted neural network compression. However, measuring latency on real devices is challenging and expensive. Therefore, this letter presents a novel and efficient framework, named EvoLP, to accurately predict the inference latency of models on edge devices. This predictor can evolve to achieve higher latency prediction precision during the network compression process. Experimental results demonstrate that EvoLP outperforms previous state-of-the-art approaches by being evaluated on three edge devices and four model variants. Moreover, when incorporated into a model compression framework, it effectively guides the compression process for higher model accuracy while satisfying strict latency constraints. We open source EvoLP at https://github.com/ntuliuteam/EvoLP.

Related papers

• Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models [0.0]
  This paper presents a benchmark of off-the-shelf generative video models for short-horizon predictive display.<n>Performance is assessed using prediction accuracy, per-rollout latency, peak GPU memory usage, and temporal error evolution.<n>Findings highlight a gap between general-purpose generative video synthesis and the requirements of predictive display in teleoperation.
  arXiv  Detail & Related papers  (2026-05-10T17:36:22Z)
• PACE: Post-Causal Entropy Modeling for Learned LiDAR Point Cloud Compression [43.524231862354405]
  LiDAR point cloud compression is vital for autonomous systems to handle massive data from high-resolution sensors.<n>We propose PACE, a new framework that reformulates ancestral context aggregation as a non-causal backbone and causality to a lightweight, stage-scalable predictor.<n>Experiments demonstrate that PACE sets a new state-of-the-art in compression efficiency, achieving notable BD-BR savings and reducing decoding latency by over 90% in autoregressive mode.
  arXiv  Detail & Related papers  (2026-05-02T08:36:11Z)
• A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits [0.08209843760716957]
  This paper presents a unified deployment-oriented comparison of static compression and dynamic early-exit mechanisms.<n>Our results show that static and dynamic techniques offer fundamentally different trade-offs for edge deployment.<n>Their combination proves highly effective, simultaneously reducing inference latency and memory usage with minimal accuracy loss.
  arXiv  Detail & Related papers  (2026-04-16T08:50:49Z)
• On Latency Predictors for Neural Architecture Search [8.564763702766776]
  We introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy.
  arXiv  Detail & Related papers  (2024-03-04T19:59:32Z)
• Overload: Latency Attacks on Object Detection for Edge Devices [47.9744734181236]
  This paper investigates latency attacks on deep learning applications. Unlike common adversarial attacks for misclassification, the goal of latency attacks is to increase the inference time. We use object detection to demonstrate how such kind of attacks work.
  arXiv  Detail & Related papers  (2023-04-11T17:24:31Z)
• Design and Prototyping Distributed CNN Inference Acceleration in Edge Computing [85.74517957717363]
  HALP accelerates inference by designing a seamless collaboration among edge devices (EDs) in Edge Computing. Experiments show that the distributed inference HALP achieves 1.7x inference acceleration for VGG-16. It is shown that the model selection with distributed inference HALP can significantly improve service reliability.
  arXiv  Detail & Related papers  (2022-11-24T19:48:30Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization [66.27399823422665]
  Device Model Generalization (DMG) is a practical yet under-investigated research topic for on-device machine learning applications.<n>We propose an efficient Device-cloUd collaborative parametErs generaTion framework DUET.
  arXiv  Detail & Related papers  (2022-09-12T13:26:26Z)
• Resource-Constrained Edge AI with Early Exit Prediction [5.060405696893342]
  We propose an early exit prediction mechanism to reduce the on-device computation overhead in a device-edge co-inference system. Specifically, we design a low-complexity module, namely the Exit Predictor, to guide some distinctly "hard" samples to bypass the computation of the early exits. Considering the varying communication bandwidth, we extend the early exit prediction mechanism for latency-aware edge inference.
  arXiv  Detail & Related papers  (2022-06-15T03:14:21Z)
• MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge [87.41163540910854]
  Deep neural network (DNN) latency characterization is a time-consuming process. We propose MAPLE-X which extends MAPLE by incorporating explicit prior knowledge of hardware devices and DNN architecture latency.
  arXiv  Detail & Related papers  (2022-05-25T11:08:20Z)
• Real-Time GPU-Accelerated Machine Learning Based Multiuser Detection for 5G and Beyond [70.81551587109833]
  nonlinear beamforming filters can significantly outperform linear approaches in stationary scenarios with massive connectivity. One of the main challenges comes from the real-time implementation of these algorithms. This paper explores the acceleration of APSM-based algorithms through massive parallelization.
  arXiv  Detail & Related papers  (2022-01-13T15:20:45Z)
• LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time [57.52251547365967]
  We propose a method for training a "compressible subspace" of neural networks that contains a fine-grained spectrum of models. We present results for achieving arbitrarily fine-grained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. Our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.
  arXiv  Detail & Related papers  (2021-10-08T17:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.