EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning

• URL: http://arxiv.org/abs/2511.19935v1
• Date: Tue, 25 Nov 2025 05:20:17 GMT
• Title: EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
• Abstract summary: EfficientXpert is a lightweight domain-pruning framework that combines a propagation-aware pruning criterion and an efficient adapter-update algorithm.<n>It enables a one-step transformation of general pretrained models into sparse, domain-adapted experts.<n>It retains up to 98% of dense-model performance at 40% sparsity, outperforming state-of-the-art methods.
• Score: 0.19116784879310023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of large language models (LLMs) has increased the demand for domain-specialized variants in areas such as law, healthcare, and finance. However, their large size remains a barrier to deployment in resource-constrained environments, and existing compression methods either generalize poorly across domains or incur high overhead. In this work, we propose \textbf{EfficientXpert}, a lightweight domain-pruning framework that combines a propagation-aware pruning criterion (Foresight Mask) with an efficient adapter-update algorithm (Partial Brain Surgeon). Integrated into the LoRA fine-tuning process, EfficientXpert enables a one-step transformation of general pretrained models into sparse, domain-adapted experts. Across health and legal tasks, it retains up to 98% of dense-model performance at 40% sparsity, outperforming state-of-the-art methods. Further analysis reveals substantial domain-dependent structural shifts that degrade the effectiveness of general pruning masks, underscoring the need for adaptive, domain-aware pruning strategies tailored to each domain.

Related papers

• BPG: Balancing Plasticity and Generalization for Domain Incremental Learning [48.68747632195046]
  Domain incremental learning enables models to continuously adapt while retaining prior knowledge.<n>We propose BPG, a unified framework that addresses both challenges through two complementary components.<n>BPG consistently outperforms uniform adapter-based approaches and hard domain selection strategies.
  arXiv  Detail & Related papers  (2026-08-11T11:22:27Z)
• Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMs [21.190823331753464]
  Vision-Language Models (VLMs) demonstrate remarkable general-purpose capabilities but often fall short in specialized domains.<n>We propose Reinforced Curriculum Pre-Alignment (RCPA), a novel post-training paradigm that introduces a curriculum-aware progressive modulation mechanism.
  arXiv  Detail & Related papers  (2026-02-11T11:04:37Z)
• RoME: Domain-Robust Mixture-of-Experts for MILP Solution Prediction across Domains [17.62400981694534]
  We introduce RoME, a domain-Robust Mixture-of-Experts framework for predicting MILP solutions across domains.<n>A single RoME model trained on three domains achieves an average improvement of 67.7%, then evaluated on five diverse domains.
  arXiv  Detail & Related papers  (2025-11-04T07:32:27Z)
• Feature-Space Planes Searcher: A Universal Domain Adaptation Framework for Interpretability and Computational Efficiency [7.889121135601528]
  Current unsupervised domain adaptation methods rely on fine-tuning feature extractors.<n>We propose Feature-space Planes Searcher (FPS) as a novel domain adaptation framework.<n>We show that FPS achieves competitive or superior performance to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-26T05:39:21Z)
• DA-Mamba: Domain Adaptive Hybrid Mamba-Transformer Based One-Stage Object Detection [0.3683202928838613]
  We present the first domain-adaptive Mamba-based one-stage object detection model, DA-Mamba.<n>Inspired by the global modeling and linear complexity of the Mamba architecture, we present the first domain-adaptive Mamba-based one-stage object detection model, DA-Mamba.
  arXiv  Detail & Related papers  (2025-02-16T15:58:54Z)
• GDO:Gradual Domain Osmosis [5.620015188968398]
  We propose a new method called Gradual Domain Osmosis, which aims to solve the problem of smooth knowledge migration from source domain to target domain in Gradual Domain Adaptation (GDA)<n>Traditional Gradual Domain Adaptation methods mitigate domain bias by introducing intermediate domains and self-training strategies, but often face the challenges of inefficient knowledge migration or missing data in intermediate domains.
  arXiv  Detail & Related papers  (2025-01-31T14:25:45Z)
• Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts [56.57141696245328]
  In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety. Existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts.
  arXiv  Detail & Related papers  (2024-11-06T11:03:02Z)
• Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shifts [133.99270341855728]
  Real-world domain styles can vary substantially due to environment changes and sensor noises. Deep models only know the training domain style. We propose Normalization Perturbation to overcome this domain style overfitting problem.
  arXiv  Detail & Related papers  (2022-11-08T17:36:49Z)
• Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation [86.02485817444216]
  We introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of 54.1% on DomainNet.
  arXiv  Detail & Related papers  (2022-09-30T03:40:10Z)
• From Big to Small: Adaptive Learning to Partial-Set Domains [94.92635970450578]
  Domain adaptation targets at knowledge acquisition and dissemination from a labeled source domain to an unlabeled target domain under distribution shift. Recent advances show that deep pre-trained models of large scale endow rich knowledge to tackle diverse downstream tasks of small scale. This paper introduces Partial Domain Adaptation (PDA), a learning paradigm that relaxes the identical class space assumption to that the source class space subsumes the target class space.
  arXiv  Detail & Related papers  (2022-03-14T07:02:45Z)
• META: Mimicking Embedding via oThers' Aggregation for Generalizable Person Re-identification [68.39849081353704]
  Domain generalizable (DG) person re-identification (ReID) aims to test across unseen domains without access to the target domain data at training time. This paper presents a new approach called Mimicking Embedding via oThers' Aggregation (META) for DG ReID.
  arXiv  Detail & Related papers  (2021-12-16T08:06:50Z)
• Domain Conditioned Adaptation Network [90.63261870610211]
  We propose a Domain Conditioned Adaptation Network (DCAN) to excite distinct convolutional channels with a domain conditioned channel attention mechanism. This is the first work to explore the domain-wise convolutional channel activation for deep DA networks.
  arXiv  Detail & Related papers  (2020-05-14T04:23:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.