RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization

• URL: http://arxiv.org/abs/2408.05752v2
• Date: Fri, 13 Sep 2024 12:37:34 GMT
• Title: RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization
• Authors: Nanyang Du, Chen Tang, Yuxiao Jiang, Yuan Meng, Zhi Wang,
• Abstract summary: We propose efficient unsupervised domain adaptation with ReTraining-Free Quantization (RTF-Q) Our approach uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic budgets. We demonstrate that our network achieves competitive accuracy with state-of-the-art methods across three benchmarks.
• Score: 14.447148108341688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Performing unsupervised domain adaptation on resource-constrained edge devices is challenging. Existing research typically adopts architecture optimization (e.g., designing slimmable networks) but requires expensive training costs. Moreover, it does not consider the considerable precision redundancy of parameters and activations. To address these limitations, we propose efficient unsupervised domain adaptation with ReTraining-Free Quantization (RTF-Q). Our approach uses low-precision quantization architectures with varying computational costs, adapting to devices with dynamic computation budgets. We subtly configure subnet dimensions and leverage weight-sharing to optimize multiple architectures within a single set of weights, enabling the use of pre-trained models from open-source repositories. Additionally, we introduce multi-bitwidth joint training and the SandwichQ rule, both of which are effective in handling multiple quantization bit-widths across subnets. Experimental results demonstrate that our network achieves competitive accuracy with state-of-the-art methods across three benchmarks while significantly reducing memory and computational costs.

Related papers

• AdaQAT: Adaptive Bit-Width Quantization-Aware Training [0.873811641236639]
  Large-scale deep neural networks (DNNs) have achieved remarkable success in many application scenarios. Model quantization is a common approach to deal with deployment constraints, but searching for optimized bit-widths can be challenging. We present Adaptive Bit-Width Quantization Aware Training (AdaQAT), a learning-based method that automatically optimize bit-widths during training for more efficient inference.
  arXiv  Detail & Related papers  (2024-04-22T09:23:56Z)
• Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
  Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs. We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
  arXiv  Detail & Related papers  (2024-03-21T02:33:37Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Efficient Controllable Multi-Task Architectures [85.76598445904374]
  We propose a multi-task model consisting of a shared encoder and task-specific decoders where both encoder and decoder channel widths are slimmable. Our key idea is to control the task importance by varying the capacities of task-specific decoders, while controlling the total computational cost. This improves overall accuracy by allowing a stronger encoder for a given budget, increases control over computational cost, and delivers high-quality slimmed sub-architectures.
  arXiv  Detail & Related papers  (2023-08-22T19:09:56Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Controllable Dynamic Multi-Task Architectures [92.74372912009127]
  We propose a controllable multi-task network that dynamically adjusts its architecture and weights to match the desired task preference as well as the resource constraints. We propose a disentangled training of two hypernetworks, by exploiting task affinity and a novel branching regularized loss, to take input preferences and accordingly predict tree-structured models with adapted weights.
  arXiv  Detail & Related papers  (2022-03-28T17:56:40Z)
• Standard Deviation-Based Quantization for Deep Neural Networks [17.495852096822894]
  Quantization of deep neural networks is a promising approach that reduces the inference cost. We propose a new framework to learn the quantization intervals (discrete values) using the knowledge of the network's weight and activation distributions. Our scheme simultaneously prunes the network's parameters and allows us to flexibly adjust the pruning ratio during the quantization process.
  arXiv  Detail & Related papers  (2022-02-24T23:33:47Z)
• Post-training Quantization for Neural Networks with Provable Guarantees [9.58246628652846]
  We modify a post-training neural-network quantization method, GPFQ, that is based on a greedy path-following mechanism. We prove that for quantizing a single-layer network, the relative square error essentially decays linearly in the number of weights.
  arXiv  Detail & Related papers  (2022-01-26T18:47:38Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Wide-band butterfly network: stable and efficient inversion via multi-frequency neural networks [1.2891210250935143]
  We introduce an end-to-end deep learning architecture called the wide-band butterfly network (WideBNet) for approximating the inverse scattering map from wide-band scattering data. This architecture incorporates tools from computational harmonic analysis, such as the butterfly factorization, and traditional multi-scale methods, such as the Cooley-Tukey FFT algorithm.
  arXiv  Detail & Related papers  (2020-11-24T21:48:43Z)
• Dataless Model Selection with the Deep Frame Potential [45.16941644841897]
  We quantify networks by their intrinsic capacity for unique and robust representations. We propose the deep frame potential: a measure of coherence that is approximately related to representation stability but has minimizers that depend only on network structure. We validate its use as a criterion for model selection and demonstrate correlation with generalization error on a variety of common residual and densely connected network architectures.
  arXiv  Detail & Related papers  (2020-03-30T23:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.