Compressing Models with Few Samples: Mimicking then Replacing

• URL: http://arxiv.org/abs/2201.02620v1
• Date: Fri, 7 Jan 2022 07:03:48 GMT
• Title: Compressing Models with Few Samples: Mimicking then Replacing
• Abstract summary: Few-sample compression aims to compress a big redundant model into a small compact one with only few samples. We propose a new framework named Mimicking then Replacing (MiR) for few-sample compression.
• Score: 33.66481681493925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-sample compression aims to compress a big redundant model into a small compact one with only few samples. If we fine-tune models with these limited few samples directly, models will be vulnerable to overfit and learn almost nothing. Hence, previous methods optimize the compressed model layer-by-layer and try to make every layer have the same outputs as the corresponding layer in the teacher model, which is cumbersome. In this paper, we propose a new framework named Mimicking then Replacing (MiR) for few-sample compression, which firstly urges the pruned model to output the same features as the teacher's in the penultimate layer, and then replaces teacher's layers before penultimate with a well-tuned compact one. Unlike previous layer-wise reconstruction methods, our MiR optimizes the entire network holistically, which is not only simple and effective, but also unsupervised and general. MiR outperforms previous methods with large margins. Codes will be available soon.

Related papers

• Understanding Layer Patching in Model Size Interpolation [30.74896485280105]
  We show that patching strongly shapes behavior, with effects that vary substantially across model families.<n>We introduce KLPatch, a greedy patching algorithm based on KL divergence, which often improves over last-to-first patching.
  arXiv  Detail & Related papers  (2026-07-09T07:14:12Z)
• TuneComp: Joint Fine-tuning and Compression for Large Foundation Models [50.33925662486034]
  sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step.<n>We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure.<n> Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods.
  arXiv  Detail & Related papers  (2025-05-27T23:49:35Z)
• GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression [26.51079570548107]
  We propose GRASP (Gradient-based Retention of Adaptive Singular Parameters), a novel compression framework.<n>By replacing redundant layers with only a minimal set of parameters, GRASP achieves efficient compression while maintaining strong performance with minimal overhead.
  arXiv  Detail & Related papers  (2024-12-31T08:22:21Z)
• Chip-Tuning: Classify Before Language Models Say [25.546473157624945]
  Chip-tuning is a simple and effective structured pruning framework for classification problems. We show that chip-tuning significantly outperforms previous state-of-the-art baselines in both accuracy and pruning ratio. We also find that chip-tuning could be applied on multimodal models, and could be combined with model finetuning, proving its excellent compatibility.
  arXiv  Detail & Related papers  (2024-10-09T04:35:22Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [66.1595537904019]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Unified Low-rank Compression Framework for Click-through Rate Prediction [15.813889566241539]
  We propose a unified low-rank decomposition framework for compressing CTR prediction models. Our framework can achieve better performance than the original model. Our framework can be applied to embedding tables and layers in various CTR prediction models.
  arXiv  Detail & Related papers  (2024-05-28T13:06:32Z)
• When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models [61.363259848264725]
  Inheritune is a simple and effective training recipe for building smaller, more efficient language models.<n>We show that Inheritune trained models, despite having significantly fewer layers, can match or even outperform their larger counterparts.
  arXiv  Detail & Related papers  (2024-04-12T17:53:34Z)
• Tiny Models are the Computational Saver for Large Models [1.8350044465969415]
  This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90%, with only negligible losses in performance.
  arXiv  Detail & Related papers  (2024-03-26T14:14:30Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• $\Delta$-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss [71.46601663956521]
  Models pre-trained on large-scale datasets are often fine-tuned to support newer tasks and datasets that arrive over time. We propose $Delta$-Patching for fine-tuning neural network models in an efficient manner, without the need to store model copies. Our experiments show that $Delta$-Networks outperform earlier model patching work while only requiring a fraction of parameters to be trained.
  arXiv  Detail & Related papers  (2023-03-26T16:39:44Z)
• Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks [55.431048995662714]
  We create a small model for a new task from the pruned models of similar tasks. We show that a few fine-tuning steps on this model suffice to produce a promising pruned-model for the new task. We develop a simple but effective ''Meta-Vote Pruning (MVP)'' method that significantly reduces the pruning iterations for a new task.
  arXiv  Detail & Related papers  (2023-01-27T06:49:47Z)
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition [7.450574974954803]
  Small memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size. We propose a multi-stage progressive approach to compress the conformer transducer model using KD.
  arXiv  Detail & Related papers  (2022-10-01T02:23:00Z)
• Structural Dropout for Model Width Compression [1.52292571922932]
  Existing ML models are known to be highly over-parametrized, and use significantly more resources than required for a given task. We propose a method that requires only a single training session for the original model and a set of compressed models. The proposed approach is a "structural" dropout that prunes all elements in the hidden state above a randomly chosen index, forcing the model to learn an importance ordering over its features.
  arXiv  Detail & Related papers  (2022-05-13T21:50:57Z)
• Prior Guided Feature Enrichment Network for Few-Shot Segmentation [64.91560451900125]
  State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results. Few-shot segmentation is proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information.
  arXiv  Detail & Related papers  (2020-08-04T10:41:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.