Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation

• URL: http://arxiv.org/abs/2605.28222v1
• Date: Wed, 27 May 2026 09:37:55 GMT
• Title: Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation
• Abstract summary: We study quality-latency-resource trade-offs in a retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator.<n>We build a benchmark of 5,144 question-answer pairs over the official documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native, Reciprocal Rank Fusion, cross-encoder reranking)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study quality-latency-resource trade-offs in a documentation-grounded retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator. We build a manually verified benchmark of 5,144 question-answer pairs over the official Kubernetes documentation and combine it with a fixed hybrid-retrieval pipeline (BGE-M3 dense, BGE-M3 native sparse, Reciprocal Rank Fusion, cross-encoder reranking). Over this benchmark we ablate 20 LoRA configurations on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct across rank and target-module choices, and evaluate each on token-level F1, LLM-judged groundedness and correctness (pass@4), inference latency, inference memory, and training cost, all reported with bootstrap 95% confidence intervals. Pareto analysis shows that LoRA adapters acting only on the q and v attention projections consistently dominate the front, while the 3B/8B choice mainly defines operating regime. A param-matched control comparison further indicates that the q/v advantage is structural rather than purely parametric. The benchmark, selected adapters, and code are available at https://github.com/EugPal/rag-lora-tradeoffs.

Related papers

• LoRA Fine-Tuned Models for Control Systems Course Q\&A: A Multidimensional Evaluation of Model Scale and Rank Effects [12.865623591533456]
  Large language models (LLMs) are increasingly used in specialized university courses.<n>Direct general-purpose responses may be inconsistently structured and hard to verify.<n>We built a supervised fine-tuning dataset of 360 system-user-assistant conversations.
  arXiv  Detail & Related papers  (2026-09-12T13:04:04Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget [72.7352550413971]
  We present Tevatron 3.0, which integrates a Megatron-Core training backend into Tevatron.<n>We show that Megatron matches FSDP reranker quality and training efficiency under comparable data-parallel settings.<n>We find that the MoE reranker matches dense 8B quality while activating less than half as many parameters and achieving substantially higher inference throughput.
  arXiv  Detail & Related papers  (2026-08-02T00:55:47Z)
• TrustMargin: Training-Free Arbitration between Parametric Memory and Retrieved Evidence in Large Language Models [10.379427318055434]
  We study this post-generation conflict as answer-level source arbitration.<n> given Direct and RAG answers from the same frozen model, decide which source to trust.<n>We propose TRUSTMARGIN, a training-free, plug-and-play arbitration layer that scores the two existing candidates with the model's own likelihoods.
  arXiv  Detail & Related papers  (2026-06-07T01:25:09Z)
• Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts [0.07812854697536452]
  We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six benchmarks.<n>We show that a substantial portion of reported unsolvability stems from evaluation artifacts.
  arXiv  Detail & Related papers  (2026-05-08T07:49:24Z)
• MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval [37.54115468116941]
  This report introduces MemReranker, a reranking model family (0.6B/4B) built on Qwen3-Reranker through multi-stage LLM knowledge distillation.<n>On the memory retrieval benchmark, MemReranker-0.6B substantially outperforms BGE-Reranker and matches open-source 4B/8B models as well as GPT-4o-mini on key metrics.<n>MemReranker-4B further achieves 0.737 MAP, with several metrics on par with Gemini-3-Flash, while maintaining inference latency at only 10-20% of large models.
  arXiv  Detail & Related papers  (2026-05-07T12:33:57Z)
• SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability [2.9388795721577328]
  Low-Rank Adaptation (LoRA) adapters are becoming a practical by-product of parameter-efficient adaptation.<n>We introduce the Sparse-Composition Agreement Layer (SCALE), a post-retrieval audit and composition framework for open-pool LoRA reuse.<n> Detailed scores, paired audits, and path-cost records are reported in the experimental section.
  arXiv  Detail & Related papers  (2026-05-02T13:00:05Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning [0.0]
  Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies.<n>This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework.
  arXiv  Detail & Related papers  (2026-03-01T15:53:49Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation [26.719288314007002]
  Large Language Models (LLMs) have transformed listwise document reranking by enabling global reasoning over candidate sets.<n>We propose textbfDeeptextbfAgenttextbfRank (textbfDeAR), an open-source framework that decouples these tasks through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-08-23T11:46:08Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs [60.38044044203333]
  Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG) We propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-turbo-2024-0409, and ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks.
  arXiv  Detail & Related papers  (2024-07-02T17:59:17Z)
• LoRA: Low-Rank Adaptation of Large Language Models [71.75808607987281]
  Low-Rank Adaptation, or LoRA, freezes the pre-trained model weights and injects trainable rank decomposition into each layer of the Transformer architecture. For GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the computation hardware requirement by 3 times compared to full fine-tuning.
  arXiv  Detail & Related papers  (2021-06-17T17:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.