Advanced Knowledge Transfer: Refined Feature Distillation for Zero-Shot Quantization in Edge Computing

• URL: http://arxiv.org/abs/2412.19125v2
• Date: Thu, 22 May 2025 07:25:40 GMT
• Title: Advanced Knowledge Transfer: Refined Feature Distillation for Zero-Shot Quantization in Edge Computing
• Authors: Inpyo Hong, Youngwan Jo, Hyojeong Lee, Sunghyun Ahn, Sanghyun Park,
• Abstract summary: AKT (Advanced Knowledge Transfer) is a novel method to enhance the training ability of low-bit quantized (Q) models.<n>Our method addresses the fundamental gradient exploding problem in low-bit Q models.
• Score: 1.8067835669244101
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce AKT (Advanced Knowledge Transfer), a novel method to enhance the training ability of low-bit quantized (Q) models in the field of zero-shot quantization (ZSQ). Existing research in ZSQ has focused on generating high-quality data from full-precision (FP) models. However, these approaches struggle with reduced learning ability in low-bit quantization due to its limited information capacity. To overcome this limitation, we propose effective training strategy compared to data generation. Particularly, we analyzed that refining feature maps in the feature distillation process is an effective way to transfer knowledge to the Q model. Based on this analysis, AKT efficiently transfer core information from the FP model to the Q model. AKT is the first approach to utilize both spatial and channel attention information in feature distillation in ZSQ. Our method addresses the fundamental gradient exploding problem in low-bit Q models. Experiments on CIFAR-10 and CIFAR-100 datasets demonstrated the effectiveness of the AKT. Our method led to significant performance enhancement in existing generative models. Notably, AKT achieved significant accuracy improvements in low-bit Q models, achieving state-of-the-art in the 3,5bit scenarios on CIFAR-10. The code is available at https://github.com/Inpyo-Hong/AKT-Advanced-knowledge-Transfer.

Related papers

• Enhancing Quantization-Aware Training on Edge Devices via Relative Entropy Coreset Selection and Cascaded Layer Correction [4.351799529311504]
  Quantization-Aware Training (QAT) on edge devices has become an effective solution.<n>Traditional QAT relies on the complete dataset for training, which incurs a huge computational cost.<n>We propose QuaRC, a QAT framework with coresets on edge devices.
  arXiv  Detail & Related papers  (2025-07-17T02:19:33Z)
• Lightweight Task-Oriented Semantic Communication Empowered by Large-Scale AI Models [66.57755931421285]
  Large-scale artificial intelligence (LAI) models pose significant challenges for real-time communication scenarios.<n>This paper proposes utilizing knowledge distillation (KD) techniques to extract and condense knowledge from LAI models.<n>We propose a fast distillation method featuring a pre-stored compression mechanism that eliminates the need for repetitive inference.
  arXiv  Detail & Related papers  (2025-06-16T08:42:16Z)
• GranQ: Granular Zero-Shot Quantization with Unified Layer-Channel Awareness [1.8067835669244101]
  GranQ is a novel ZSQ approach that leverages layer-channel awareness to minimize the quantization error. GranQ achieves superior performance compared with those of state-of-the-art ZSQ methods that employ quantization-aware training.
  arXiv  Detail & Related papers  (2025-03-24T04:44:21Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG.<n>Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization.<n>Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• Quantized and Interpretable Learning Scheme for Deep Neural Networks in Classification Task [0.0]
  We introduce an approach that combines saliency-guided training with quantization techniques to create an interpretable and resource-efficient model.<n>Our results demonstrate that the combined use of saliency-guided training and PACT-based quantization not only maintains classification performance but also produces models that are significantly more efficient and interpretable.
  arXiv  Detail & Related papers  (2024-12-05T06:34:06Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [66.1595537904019]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models [88.80146574509195]
  Quantization is a promising approach for reducing memory overhead and accelerating inference. We propose a novel-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs.
  arXiv  Detail & Related papers  (2023-10-20T07:09:56Z)
• EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization [3.3998740964877463]
  Quantization is a key method for deploying deep neural networks on edge devices with limited memory and computation resources. We propose a new method for enhanced Post-Training Quantization (EPTQ) that employs a network-wise quantization optimization process.
  arXiv  Detail & Related papers  (2023-09-20T10:50:28Z)
• Online Hyperparameter Optimization for Class-Incremental Learning [99.70569355681174]
  Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phase-by-phase. An inherent challenge of CIL is the stability-plasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. We propose an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori.
  arXiv  Detail & Related papers  (2023-01-11T17:58:51Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• Encoding Explanatory Knowledge for Zero-shot Science Question Answering [0.755972004983746]
  N-XKT is able to improve accuracy and generalization on science Question Answering (QA) N-XKT model shows a clear improvement on zero-shot QA. A systematic analysis is conducted to quantitatively analyze the performance of the N-XKT model.
  arXiv  Detail & Related papers  (2021-05-12T15:42:50Z)
• Zero-shot Adversarial Quantization [11.722728148523366]
  We propose a zero-shot adversarial quantization (ZAQ) framework, facilitating effective discrepancy estimation and knowledge transfer. This is achieved by a novel two-level discrepancy modeling to drive a generator to synthesize informative and diverse data examples. We conduct extensive experiments on three fundamental vision tasks, demonstrating the superiority of ZAQ over the strong zero-shot baselines.
  arXiv  Detail & Related papers  (2021-03-29T01:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.