SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization

• URL: http://arxiv.org/abs/2604.08368v1
• Date: Thu, 09 Apr 2026 15:34:13 GMT
• Title: SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization
• Abstract summary: SOLAR is a post-training compression framework that substantially reduces the communication cost of PEFT adapters.<n>It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules.
• Score: 21.389930928202535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, enable scalable adaptation of foundation models by injecting low-rank adapters. However, their communication and storage costs remain a major bottleneck in resource-constrained settings. We propose SOLAR (Subspace-Oriented Latent Adapter Reparameterization), a post-training compression framework that substantially reduces the communication cost (i.e., the number of parameters to transmit or store) of PEFT adapters. SOLAR expresses each PEFT update as a linear combination of basis vectors formed from the foundation model's singular vectors with controlled random perturbations. By exploiting the subspace similarity (the alignment of principal directions) between the foundation model and task-specific fine-tuned updates, SOLAR decouples the adapter size from PEFT structure and ensures compact yet expressive representations. It is model-agnostic and compatible with existing PEFT methods, including LoRA, AdaLoRA, and other adapter modules. We theoretically establish a bound on the reconstruction error. Experiments on language and vision tasks using LLaMA, GPT, and ViT models demonstrate that SOLAR preserves task performance while significantly reducing model representation sizes, offering an effective and communication-efficient solution for deployment in distributed systems and edge devices.

Related papers

• Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing [16.45880716714667]
  Domain-aware Relaxed Orthogonal Subspace adaptation (DROS)<n>DROS reformulates low-rank adaptation as data-conditioned subspace learning and flexible subspace adaptation.<n>Experiments on multiple remote sensing benchmarks demonstrate that DROS achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-09-12T02:19:11Z)
• Improving Recursive Transformers with Mixture of LoRAs [2.672804414228544]
  Mixture of LoRAs (MoL) inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN)<n>MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters.<n>ModernALBERT achieves state-of-the-art performance among compact models.
  arXiv  Detail & Related papers  (2025-12-14T23:39:30Z)
• TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning [83.93651411533533]
  We introduce Tucker Adaptation (TuckA), a method with four key properties.<n>We develop an efficient batch-level routing mechanism, which reduces the router's parameter size by a factor of $L$.<n>Experiments on benchmarks in natural language understanding, image classification, and mathematical reasoning speak to the efficacy of TuckA.
  arXiv  Detail & Related papers  (2025-11-10T09:03:16Z)
• Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts [72.22148263683037]
  We study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures.<n>First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature.<n>Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks.
  arXiv  Detail & Related papers  (2025-07-09T03:25:45Z)
• Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models [13.742950928229078]
  Low-Rank Adaptation (LoRA) addresses these issues by training compact, low-rank matrices instead of fully fine-tuning large models.<n>This paper introduces a wireless federated LoRA fine-tuning framework that optimize both learning performance and communication efficiency.
  arXiv  Detail & Related papers  (2025-05-01T06:15:38Z)
• QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuning [0.0]
  Scaling full finetuning of large foundation models strains GPU memory and training time.<n>We introduce Quantum-Inspired Compound Adapters (QuIC Adapters)<n>QuIC adapters can effectively finetune a model using less than 0.02% memory footprint of the base model.
  arXiv  Detail & Related papers  (2025-02-10T13:06:56Z)
• LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation [48.22550575107633]
  A new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), enables the training-free transfer of LoRA parameters across source and target models.<n>Our experiments demonstrate the effectiveness of LoRA-X for text-to-image generation.
  arXiv  Detail & Related papers  (2025-01-27T23:02:24Z)
• Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model [81.55141188169621]
  We equip PEFT with a cross-block orchestration mechanism to enable the adaptation of the Segment Anything Model (SAM) to various downstream scenarios. We propose an intra-block enhancement module, which introduces a linear projection head whose weights are generated from a hyper-complex layer. Our proposed approach consistently improves the segmentation performance significantly on novel scenarios with only around 1K additional parameters.
  arXiv  Detail & Related papers  (2023-11-28T11:23:34Z)
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning [112.97430455461097]
  We propose a general PEFT method that tunes a mixture of adaptation modules introduced in each Transformer layer while keeping most of the PLM weights frozen. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.
  arXiv  Detail & Related papers  (2022-10-31T16:23:36Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.