Pruning General Large Language Models into Customized Expert Models

• URL: http://arxiv.org/abs/2506.02561v1
• Date: Tue, 03 Jun 2025 07:47:30 GMT
• Title: Pruning General Large Language Models into Customized Expert Models
• Authors: Yirao Zhao, Guizhen Chen, Kenji Kawaguchi, Lidong Bing, Wenxuan Zhang,
• Abstract summary: Large language models (LLMs) have revolutionized natural language processing, yet their substantial model sizes often require substantial computational resources.<n>We propose a method to prune a large general model into a smaller lightweight expert model.<n>By identifying and pruning irrelevant neurons of each dimension, $textttCus-Prun$ creates expert models without any post-training.
• Score: 78.96891010334852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have revolutionized natural language processing, yet their substantial model sizes often require substantial computational resources. To preserve computing resources and accelerate inference speed, it is crucial to prune redundant parameters, especially for experienced users who often need compact expert models tailored to specific downstream scenarios. However, most existing pruning methods focus on preserving the model's general capabilities, often requiring extensive post-training or suffering from degraded performance due to coarse-grained pruning. In this work, we design a $\underline{Cus}$tom $\underline{Prun}$ing method ($\texttt{Cus-Prun}$) to prune a large general model into a smaller lightweight expert model, which is positioned along the "language", "domain" and "task" dimensions. By identifying and pruning irrelevant neurons of each dimension, $\texttt{Cus-Prun}$ creates expert models without any post-training. Our experiments demonstrate that $\texttt{Cus-Prun}$ consistently outperforms other methods, achieving minimal loss in both expert and general capabilities across various models from different model families and sizes.

Related papers

• GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching [41.96482857947199]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation.<n>LLMs typically come with a substantial model size, which presents significant challenges in deployment and inference.<n>We develop a novel strategy to compress models by strategically combining or merging layers from finetuned model variants.
  arXiv  Detail & Related papers  (2025-06-25T14:24:59Z)
• Intention-Conditioned Flow Occupancy Models [69.79049994662591]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models [64.18350535770357]
  We propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning.<n>Our approach only leverages a small number of samples to search for the desired pruning policy.<n>We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering.
  arXiv  Detail & Related papers  (2025-03-19T16:07:04Z)
• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• LaCo: Large Language Model Pruning via Layer Collapse [56.92068213969036]
  Large language models (LLMs) based on transformer are witnessing a notable trend of size expansion. Existing methods such as model quantization, knowledge distillation, and model pruning are constrained by various issues. We propose a concise layer-wise structured pruner called textitLayer Collapse (LaCo), in which rear model layers collapse into a prior layer.
  arXiv  Detail & Related papers  (2024-02-17T04:16:30Z)
• Model Compression and Efficient Inference for Large Language Models: A Survey [20.199282252344396]
  Large language models have two prominent characteristics compared to smaller models. The most notable aspect of large models is the very high cost associated with model finetuning or training. Large models emphasize versatility and generalization rather than performance on a single task.
  arXiv  Detail & Related papers  (2024-02-15T06:58:30Z)
• Selective Pre-training for Private Fine-tuning [33.55628974557588]
  We show that a careful pre-training on a public dataset is crucial to train small language models with differential privacy. Results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data.
  arXiv  Detail & Related papers  (2023-05-23T09:36:58Z)
• Structural Dropout for Model Width Compression [1.52292571922932]
  Existing ML models are known to be highly over-parametrized, and use significantly more resources than required for a given task. We propose a method that requires only a single training session for the original model and a set of compressed models. The proposed approach is a "structural" dropout that prunes all elements in the hidden state above a randomly chosen index, forcing the model to learn an importance ordering over its features.
  arXiv  Detail & Related papers  (2022-05-13T21:50:57Z)
• TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models [18.49325959450621]
  We introduce TextPruner, an open-source model pruning toolkit for pre-trained language models. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model.
  arXiv  Detail & Related papers  (2022-03-30T02:10:33Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.