Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression

• URL: http://arxiv.org/abs/2603.27383v1
• Date: Sat, 28 Mar 2026 19:29:38 GMT
• Title: Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression
• Abstract summary: We present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (CRISP)<n>CRISP seamlessly integrates multiple PR tasks within the same framework.<n> Experiments show CRISP outperforms methods from prior work capable of dual-task applications by 4-5%.
• Score: 30.925082859761215
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Parameter Recombination (PR) methods aim to efficiently compose the weights of a neural network for applications like Parameter-Efficient FineTuning (PEFT) and Model Compression (MC), among others. Most methods typically focus on one application of PR, which can make composing them challenging. For example, when deploying a large model you may wish to compress the model and also quickly adapt to new settings. However, PEFT methods often can still contain millions of parameters. This may be small compared to the original model size, but can be problematic in resource constrained deployments like edge devices, where they take a larger portion of the compressed model's parameters. To address this, we present Coefficient-gated weight Recombination by Interpolated Shared basis Projections (CRISP), a general approach that seamlessly integrates multiple PR tasks within the same framework. CRISP accomplishes this by factorizing pretrained weights into basis matrices and their component mixing projections. Sharing basis matrices across layers and adjusting its size enables us to perform MC, whereas the mixer weight's small size (fewer than 200 in some experiments) enables CRISP to support PEFT. Experiments show CRISP outperforms methods from prior work capable of dual-task applications by 4-5\% while also outperforming the state-of-the-art in PEFT by 1.5\% and PEFT+MC combinations by 1\%. Our code is available on the repository: https://github.com/appledora/CRISP-CVPR26.

Related papers

• Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs [50.677556944160216]
  MoEXBench is a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow.<n>It measures composable-compression quality, workload and architecture, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware.
  arXiv  Detail & Related papers  (2026-08-22T00:01:18Z)
• Plug-and-Play Parameter-Efficient Tuning of Embeddings for Federated Recommendation [16.783108944320066]
  Federated Recommendation (FR) enables such multi-end collaborative training while preserving privacy by sharing model parameters instead of raw data.<n>We propose a FR training framework with Fine-Tuning (PEFT) based embedding designed to reduce the volume of embedding parameters that need to be transmitted.<n>In addition to incorporating common PEFT techniques such as LoRA and Hash-based encoding, we explore the use of Residual Quantized Variational Autoencoders (RQ-VAE) as a novel PEFT strategy within our framework.
  arXiv  Detail & Related papers  (2025-12-14T07:38:49Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks [16.512587987753967]
  RECAST is a novel method that dramatically reduces task-specific trainable parameters to fewer than 50.<n>We show that RECAST outperforms the state-of-the-art by up to 3% across various scales, architectures, and parameter spaces.
  arXiv  Detail & Related papers  (2024-11-25T19:08:38Z)
• Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation [33.05581803204543]
  Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size.<n>We introduce SketchTune, a compressive adaptation strategy that compresses weights into compact fine-tunable sketches.<n>SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods.
  arXiv  Detail & Related papers  (2024-10-08T20:58:24Z)
• Learning to Compose SuperWeights for Neural Parameter Allocation Search [61.078949532440724]
  We show that our approach can generate parameters for many network using the same set of weights. This enables us to support tasks like efficient ensembling and anytime prediction.
  arXiv  Detail & Related papers  (2023-12-03T04:20:02Z)
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization [92.36876698509164]
  We present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models.<n>In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x.
  arXiv  Detail & Related papers  (2023-11-22T05:28:59Z)
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning [112.97430455461097]
  We propose a general PEFT method that tunes a mixture of adaptation modules introduced in each Transformer layer while keeping most of the PLM weights frozen. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.
  arXiv  Detail & Related papers  (2022-10-31T16:23:36Z)
• Parameter-Efficient Sparsity for Large Language Models Fine-Tuning [63.321205487234074]
  We propose a. sparse-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training. Experiments with diverse networks (i.e., BERT, RoBERTa and GPT-2) demonstrate PST performs on par or better than previous sparsity methods.
  arXiv  Detail & Related papers  (2022-05-23T02:43:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.