Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

• URL: http://arxiv.org/abs/2605.24058v1
• Date: Fri, 22 May 2026 02:37:14 GMT
• Title: Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
• Abstract summary: We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers.<n>A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors.
• Score: 5.790458475928127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot-swapping.We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers while representing magnitudes through lightweight, channel-wise scales, converting the dense adapter branch into two sign-accumulation matrix multiplications interleaved with channel-wise scaling. A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors. In adapter-mode experiments, LoRDBA outperforms low-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes. The unmerged adapter incurs at most 8% prefill latency overhead at matched rank r=16 despite an over 10x reduction in adapter footprint, with moderate training memory overhead of approximately 1.6x that of fp16 LoRA.

Related papers

• LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution [73.85089704122315]
  We propose a minimalist framework that performs Video Super-Resolution (VSR) using a frozen Transformer with a lightweight State-Aware Adapter.<n> LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.
  arXiv  Detail & Related papers  (2026-06-08T09:23:42Z)
• Rethinking Adapter Placement: A Dominant Adaptation Module Perspective [14.452224342067657]
  Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning method that places low-rank adapters into frozen pre-trained models.<n>Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly.<n>We propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module.
  arXiv  Detail & Related papers  (2026-05-07T13:01:00Z)
• Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models [19.288371639304504]
  Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs.<n>We introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning.
  arXiv  Detail & Related papers  (2026-01-08T20:34:12Z)
• Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems [11.584593298674688]
  Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs)<n>In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model.<n>We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving.
  arXiv  Detail & Related papers  (2025-11-28T05:04:02Z)
• MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting [6.335488846185043]
  MSLoRA is a backbone-agnostic, parameter-efficient adapter that reweights feature responses rather than re-tuning the underlying backbone.<n>MSLoRA unifies adaptation for both convolutional neural networks (CNNs) and vision transformers (ViTs)
  arXiv  Detail & Related papers  (2025-11-16T00:35:37Z)
• Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts [72.22148263683037]
  We study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures.<n>First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature.<n>Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks.
  arXiv  Detail & Related papers  (2025-07-09T03:25:45Z)
• Low-rank variational dropout: Uncertainty and rank selection in adapters [0.0]
  BayesLoRA revisits variational dropout in the LoRA setting and shows that the natural unit ofity is not individual weights but entire ranks of the adapter.<n>By placing rank-wise variational distributions over adapter components, BayesLoRA defines a posterior that (i) yields predictions through adapter-only Monte Carlo sampling and (ii) prunes redundant ranks automatically via an ARD-style KL term.
  arXiv  Detail & Related papers  (2025-06-28T08:22:02Z)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models [54.1231638555233]
  Low-rank Sparse Adaptation (LoSA) is a novel method that seamlessly integrates low-rank adaptation into sparse LLM sparsity.<n>LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning.<n>LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden.
  arXiv  Detail & Related papers  (2025-02-20T18:37:32Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters [16.160749645651567]
  We propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion.
  arXiv  Detail & Related papers  (2024-07-22T22:46:36Z)
• LCM-LoRA: A Universal Stable-Diffusion Acceleration Module [52.8517132452467]
  Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks. This report further extends LCMs' potential by applying LoRA distillation to larger Stable-Diffusion models. We identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA.
  arXiv  Detail & Related papers  (2023-11-09T18:04:15Z)
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters [59.490751234925206]
  Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks. We present S-LoRA, a system designed for the scalable serving of many LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-06T17:26:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.