SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability

• URL: http://arxiv.org/abs/2605.01429v1
• Date: Sat, 02 May 2026 13:00:05 GMT
• Title: SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
• Abstract summary: Low-Rank Adaptation (LoRA) adapters are becoming a practical by-product of parameter-efficient adaptation.<n>We introduce the Sparse-Composition Agreement Layer (SCALE), a post-retrieval audit and composition framework for open-pool LoRA reuse.<n> Detailed scores, paired audits, and path-cost records are reported in the experimental section.
• Score: 2.9388795721577328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Libraries of Low-Rank Adaptation (LoRA) adapters are becoming a practical by-product of parameter-efficient adaptation. Once such adapters accumulate, a natural question is no longer how to train one adapter for one task, but how to reuse an open pool of adapters for a new task given only a small support set. Prior work has shown that LoRA modules can be composed at the task level and dynamically selected at the instance level. However, open-pool LoRA reuse is not automatic: retrieving relevant adapters does not guarantee that their parameter updates are compatible, and composing adapters does not guarantee reliable outputs. We introduce the Sparse-Composition Agreement Layer (SCALE), a post-retrieval audit and composition framework for open-pool LoRA reuse. SCALE contains a deployable 1.0* merge path, Layer-Adaptive Sparse Residual Composition (LASRC), and a higher-cost reliability-analysis layer for multi-view disagreement. LASRC addresses merge interference by preserving a linear anchor while residualizing block-wise adapter update directions. The reliability layer treats disagreement among sparse composition views as an observable uncertainty signal and compares agreement, support-loss proxy selection, and oracle headroom under explicit path cost. In matched FLAN-T5-Large, BIG-Bench Hard (BBH), and 97-LoRA experiments, LASRC gives a directional single-view gain under fixed retrieval, while SCALE-support is reported as a query-label-free 3.0* reliability-analysis variant rather than as a calibrated or throughput-equivalent selector. Protocol-distinct BBH-8 validation shows the same qualitative trend on three decoder-only backbones. Detailed scores, paired audits, and path-cost records are reported in the experimental section.

Related papers

• LiST: Local-Simplex Test-Time LoRA Fusion [68.96696100246577]
  textbfLiST is a label-free test-time LoRA fusion framework.<n>Experiments on multimodal and language benchmarks show that LiST outperforms static LoRA merging and conventional test-time adaptation baselines.
  arXiv  Detail & Related papers  (2026-08-23T11:24:28Z)
• Empowering Compact LLMs with Fusion of Layer-wise Exits for Recommendation [59.216721553953]
  Large language model-based recommender systems (LLM-RSs) have demonstrated remarkable capabilities, but are computationally unsustainable for many real-world applications.<n>We propose a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking.<n>Experiments on three real-world datasets with Qwen 3 1.7B and Llama 3.2 3B show that FLEXRec achieves state-of-the-art accuracy while remaining highly efficient.
  arXiv  Detail & Related papers  (2026-08-18T03:14:28Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling [56.888597798431704]
  Open-World Object Detection (OWOD) requires detectors to identify previously unseen objects as unknown and incrementally incorporate them into the set of known categories.<n>Existing frameworks rely heavily on exemplar replay to mitigate catastrophic forgetting, but in some real applications, storing raw data conflicts with data access restrictions and leads to data exposure risks.<n>We propose REAL-OW, a novel rehearsal-free framework that decouples incremental knowledge through a collaborative adapter architecture.
  arXiv  Detail & Related papers  (2026-07-03T06:38:54Z)
• Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation [0.0]
  The benchmark introduced here contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted labels under a fixed schema.<n>The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting.
  arXiv  Detail & Related papers  (2026-06-27T13:59:46Z)
• Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation [0.0]
  We study quality-latency-resource trade-offs in a retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator.<n>We build a benchmark of 5,144 question-answer pairs over the official documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native, Reciprocal Rank Fusion, cross-encoder reranking)
  arXiv  Detail & Related papers  (2026-05-27T09:37:55Z)
• Relatron: Automating Relational Machine Learning over Relational Databases [50.94254514286021]
  We present a study that unifies RDL and DFS in a shared design space and conducts architecture-centric searches across diverse RDB tasks.<n>Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and accuracy is an unreliable guide for choice architecture.
  arXiv  Detail & Related papers  (2026-02-26T02:45:22Z)
• The Appeal and Reality of Recycling LoRAs with Adaptive Merging [33.0501514389443]
  We consider recycling LoRAs found "in the wild" on model repositories like the Hugging Face Hub.<n>We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA.<n>This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect.
  arXiv  Detail & Related papers  (2026-02-12T18:59:20Z)
• Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases [3.4869850730657728]
  We present a novel framework for dynamic LoRA adapter composition that leverages similarity retrieval in vector databases.<n>Our approach constructs a task-aware vector database by embedding training examples from 22 spanning commonsense reasoning, question answering, natural language inference, and sentiment analysis.<n>Our framework requires no additional retriever training, operates with frozen embeddings, and enables efficient, interpretable adapter composition.
  arXiv  Detail & Related papers  (2026-02-01T22:20:04Z)
• Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters [27.694134466842502]
  Parametric Retrieval-Augmented Generation (PRAG) integrates external knowledge directly into a Large Language Model (LLM)<n>Current PRAG approaches adopt a textbfone-to-one document encoding scheme, using a dedicated LoRA adapter for each individual document.<n>We propose a novel paradigm for encoding passages in PRAG that utilizes a latent routing encoding process.
  arXiv  Detail & Related papers  (2025-11-21T08:44:21Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• Tensorized Clustered LoRA Merging for Multi-Task Interference [29.488565631016332]
  In multi-task settings, merging LoRA adapters trained on heterogeneous sources frequently causes textittask interference, degrading downstream performance.<n>We propose a tensorized clustered LoRA library targeting to address the task interference.<n> TC-LoRA achieves +1.4% accuracy on Phi-3 and +2.3% on Mistral-7B, demonstrating the effectiveness of TC-LoRA in LLM adaptation.
  arXiv  Detail & Related papers  (2025-08-06T01:26:43Z)
• Low-rank variational dropout: Uncertainty and rank selection in adapters [0.0]
  BayesLoRA revisits variational dropout in the LoRA setting and shows that the natural unit ofity is not individual weights but entire ranks of the adapter.<n>By placing rank-wise variational distributions over adapter components, BayesLoRA defines a posterior that (i) yields predictions through adapter-only Monte Carlo sampling and (ii) prunes redundant ranks automatically via an ARD-style KL term.
  arXiv  Detail & Related papers  (2025-06-28T08:22:02Z)
• Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning [57.36978335727009]
  Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs) In this paper, we propose a framework that adaptively retrieves and composes multiple LoRAs based on input prompts.
  arXiv  Detail & Related papers  (2024-06-24T05:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.