Related papers: SplitLite: Low-Rank Residual Compression for Split Learning

SplitLite: Low-Rank Residual Compression for Split Learning

URL: http://arxiv.org/abs/2608.23018v1
Date: Mon, 24 Aug 2026 09:20:04 GMT
Title: SplitLite: Low-Rank Residual Compression for Split Learning
Abstract summary: SplitLite is a communication-efficient split federated LoRA fine-tuning method.<n>By revealing this property, SplitLite transmits only quantized truncated singular value decomposition residual factors.<n>Our method reduces activation uplink communication costs by up to 93.5% and total communication costs by up to 83.7%, without performance degradation.
Score: 13.143425367328385
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, when LoRA uses rank $r$ updates in parameter space, the activation and gradient residuals of the same data sample between adjacent epochs also exhibit effective rank-$2r$ and rank-$4r$ structures, respectively. By revealing this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 83.7\%, without performance degradation.

Related papers

FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks [28.49324627841803]
Federated learning enables fine-tuning of large language models without sharing raw client data.<n>We propose FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks.<n>Our proposed framework reduces communication overhead by 50-65% while achieving rapid convergence to competitive perplexity compared to baselines.
arXiv Detail & Related papers (2026-09-11T22:37:39Z)
LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections [54.51153206716027]
Split learning (SL) enables collaborative training by partitioning a neural network across clients and a central server.<n>We propose LightSplit, which limits information exposure and reduces communication overhead by applying a lightweight fixed random projection at the cut layer.<n>Our results show that the method retains more than 95% of the baseline accuracy at up to 32x reduction in transmitted dimensionality.
arXiv Detail & Related papers (2026-05-13T09:45:49Z)
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC [50.36542772932594]
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning.<n>We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent.
arXiv Detail & Related papers (2026-02-18T13:41:41Z)
Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
arXiv Detail & Related papers (2026-01-30T10:54:51Z)
How to Train Your LLM Web Agent: A Statistical Diagnosis [96.86317871461834]
We present the first statistically grounded study on compute allocation for LLM web-agent post-training.<n>Our approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate a Llama 3.3 70B teacher via supervised fine-tuning (SFT) and on-policy reinforcement learning.<n>Our results show that combining SFT with on-policy RL consistently outperforms either approach alone on both WorkArena and MiniWob++.
arXiv Detail & Related papers (2025-07-05T17:12:33Z)
HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models [30.345920952847752]
Large language models (LLMs) have achieved remarkable breakthroughs, revolutionizing the natural language processing domain and beyond.<n>Due to immense parameter sizes, fine-tuning these models with private data for diverse downstream tasks has become mainstream.<n>We propose HSplitLoRA, a framework built on split learning (SL) and low-rank adaptation (LoRA) fine-tuning, for efficiently fine-tuning LLMs on heterogeneous client devices.
arXiv Detail & Related papers (2025-05-05T17:09:19Z)
SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices [15.790762116995845]
Fine-tuning large language models (LLMs) on private, on-device data can empower tailored personalized AI agents.<n>This paper proposes SplitFrozen, a split learning framework that enables efficient fine-tuning on resource-constrained edge devices.<n> Experiments on GPT-2 with the MRPC, MNLI-matched, and SST-2 datasets demonstrate that SplitFrozen outperforms FedLoRA and SplitLoRA by 69.4% model accuracy under extremely imbalanced data.
arXiv Detail & Related papers (2025-03-23T08:03:44Z)
Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
arXiv Detail & Related papers (2024-11-01T14:28:39Z)
SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
arXiv Detail & Related papers (2024-07-10T10:22:35Z)
LW-FedSSL: Resource-efficient Layer-wise Federated Self-supervised Learning [14.937324556028548]
We propose the Layer-Wise Federated Self-Supervised Learning approach, which allows edge devices to incrementally train a small part of the model at a time.<n>Since only a portion of the model is active for training at any given time, LW-FedSSL significantly reduces computational requirements.<n>It can achieve up to a $3.34 times$ reduction in memory usage, $4.20 times$ fewer computational operations (GFLOPs), and a $5.07 times$ lower communication cost.
arXiv Detail & Related papers (2024-01-22T01:57:31Z)
Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks [44.37047471448793]
In this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL) We propose an innovative PSL framework, namely, efficient parallel split learning (EPSL) to accelerate model training. We show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy.
arXiv Detail & Related papers (2023-03-26T16:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.