HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices

• URL: http://arxiv.org/abs/2605.05819v1
• Date: Thu, 07 May 2026 07:57:23 GMT
• Title: HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
• Abstract summary: We propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU.<n> Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks.
• Score: 10.436245372331003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, they often suffer from substantial accuracy degradation or severe throughput bottlenecks. Recent error compensation methods recover accuracy through auxiliary LoRA-style branches, and we observe that these branches are inherently amenable to offloading: they require substantial parameter storage but access only a small subset of compensation parameters during each inference step. Motivated by this opportunity, we propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU, and further introduces an asynchronous compensation pipeline and sensitivity-aware dynamic rank allocation to hide compensation overhead and maximize accuracy recovery. Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks compared to compression model and sustaining a maximum speedup of 10.4x compared to full-precision model.

Related papers

• SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving [26.96887030437247]
  We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving.<n> SPEAR introduces lightweight Error Compensators (ECs) by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic.<n>We show that SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.
  arXiv  Detail & Related papers  (2026-06-04T22:38:53Z)
• PACE: Post-Causal Entropy Modeling for Learned LiDAR Point Cloud Compression [43.524231862354405]
  LiDAR point cloud compression is vital for autonomous systems to handle massive data from high-resolution sensors.<n>We propose PACE, a new framework that reformulates ancestral context aggregation as a non-causal backbone and causality to a lightweight, stage-scalable predictor.<n>Experiments demonstrate that PACE sets a new state-of-the-art in compression efficiency, achieving notable BD-BR savings and reducing decoding latency by over 90% in autoregressive mode.
  arXiv  Detail & Related papers  (2026-05-02T08:36:11Z)
• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning [74.5532558466687]
  Group Relative Reward Rescaling (GR$3$) is a general, continuous, and reward-dependent gating mechanism.<n> GR$3$maintains training dynamics and downstream performance comparable to standard GRPO.<n>It significantly mitigates length inflation, outperforming state-of-the-art length-regularized baselines.
  arXiv  Detail & Related papers  (2026-03-11T08:41:34Z)
• SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression [27.258302662888166]
  SAES-SVD is a low-rank compression framework for large language models.<n>It jointly optimize intra-layer reconstruction and inter-layer error compensation.<n>Experiments show that SAES-SVD consistently improves post-compression performance.
  arXiv  Detail & Related papers  (2026-02-03T03:23:10Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation [11.078693613992556]
  We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation.<n>Our method delivers a superior bandwidth-accuracy trade-off and improved throughput.
  arXiv  Detail & Related papers  (2025-12-18T21:15:54Z)
• QuantVSR: Low-Bit Post-Training Quantization for Real-World Video Super-Resolution [53.13952833016505]
  We propose a low-bit quantization model for real-world video super-resolution (VSR)<n>We use a calibration dataset to measure both spatial and temporal complexity for each layer.<n>We refine the FP and low-bit branches to achieve simultaneous optimization.
  arXiv  Detail & Related papers  (2025-08-06T14:35:59Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration [34.43633070396096]
  State-Space Models (SSMs) have attracted considerable attention in Image Restoration (IR)<n>Q-MambaIR is an accurate, efficient, and flexible Quantized Mamba for IR tasks.
  arXiv  Detail & Related papers  (2025-03-27T20:34:11Z)
• EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  EoRA is a fine-tuning-free method that augments compressed Large Language Models with low-rank matrices.<n>EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• Temporal Feature Matters: A Framework for Diffusion Model Quantization [105.3033493564844]
  Diffusion models rely on the time-step for the multi-round denoising.<n>We introduce a novel quantization framework that includes three strategies.<n>This framework preserves most of the temporal information and ensures high-quality end-to-end generation.
  arXiv  Detail & Related papers  (2024-07-28T17:46:15Z)
• Communication-Efficient Distributed Learning with Local Immediate Error Compensation [95.6828475028581]
  We propose the Local Immediate Error Compensated SGD (LIEC-SGD) optimization algorithm. LIEC-SGD is superior to previous works in either the convergence rate or the communication cost.
  arXiv  Detail & Related papers  (2024-02-19T05:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.