Optimizing Retrieval-Augmented Generation (RAG) for Colloquial Cantonese: A LoRA-Based Systematic Review

• URL: http://arxiv.org/abs/2508.08610v1
• Date: Tue, 12 Aug 2025 03:46:16 GMT
• Title: Optimizing Retrieval-Augmented Generation (RAG) for Colloquial Cantonese: A LoRA-Based Systematic Review
• Authors: David Santandreu Calonge, Linda Smail,
• Abstract summary: Review examines advances in.<n>Efficient Fine-Tuning (PEFT) to optimize Retrieval-Augmented Generation (RAG) systems like Qwen3, DeepSeek, and Kimi.<n>RAG systems face challenges in understanding and generating authentic Cantonese colloquial expressions due to limited annotated data and linguistic variability.<n>Findings reveal that dynamic and ensemble LoRA adaptations significantly reduce trainable parameters without sacrificing retrieval accuracy and generation quality in dialectal contexts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This review examines recent advances in Parameter-Efficient Fine-Tuning (PEFT), with a focus on Low-Rank Adaptation (LoRA), to optimize Retrieval-Augmented Generation (RAG) systems like Qwen3, DeepSeek, and Kimi. These systems face challenges in understanding and generating authentic Cantonese colloquial expressions due to limited annotated data and linguistic variability. The review evaluates the integration of LoRA within RAG frameworks, benchmarks PEFT methods for retrieval and generation accuracy, identify domain adaptation strategies under limited data, and compares fine-tuning techniques aimed at improving semantic fidelity under data-scarce conditions. A systematic analysis of recent studies employing diverse LoRA variants, synthetic data generation, user feedback integration, and adaptive parameter allocation was conducted to assess their impact on computational efficiency, retrieval precision, linguistic authenticity, and scalability. Findings reveal that dynamic and ensemble LoRA adaptations significantly reduce trainable parameters without sacrificing retrieval accuracy and generation quality in dialectal contexts. However, limitations remain in fully preserving fine-grained linguistic nuances, especially for low-resource settings like Cantonese. The integration of real-time user feedback and domain-specific data remains underdeveloped, limiting model adaptability and personalization. While selective parameter freezing and nonlinear adaptation methods offer better trade-offs between efficiency and accuracy, their robustness at scale remains an open challenge. This review highlights the promise of PEFT-enhanced RAG systems for domain-specific language tasks and calls for future work targeting dialectal authenticity, dynamic adaptation, and scalable fine-tuning pipelines.

Related papers

• GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASR [11.705969106735454]
  GLoRIA is an adaptation framework that modulates low-rank updates in a pre-trained encoder.<n>On the GCND corpus, GLoRIA outperforms geo-conditioned full fine-tuning.
  arXiv  Detail & Related papers  (2026-03-02T23:10:09Z)
• Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages [11.808922632545874]
  We analyze multilingual automatic speech recognition models and reveal a U-shaped adaptability pattern.<n>We propose DAMA, a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer's role.<n>Dama matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters, achieves a 29% error reduction under extreme data scarcity, and significantly improves memory, training time, and computational efficiency over baselines.
  arXiv  Detail & Related papers  (2026-02-01T04:18:31Z)
• Predict the Retrieval! Test time adaptation for Retrieval Augmented Generation [66.36556189794526]
  TTARAG is a test-time adaptation method that dynamically updates the language model's parameters during inference to improve RAG system performance in specialized domains.<n>Our method introduces a simple yet effective approach where the model learns to predict retrieved content, enabling automatic parameter adjustment to the target domain.
  arXiv  Detail & Related papers  (2026-01-16T17:07:01Z)
• Influence Guided Context Selection for Effective Retrieval-Augmented Generation [23.188397777606095]
  Retrieval-Augmented Generation (RAG) addresses large language model (LLM) hallucinations by grounding responses in external knowledge.<n>Existing approaches attempt to improve performance through context selection based on predefined context quality assessment metrics.<n>We reconceptualize context quality assessment as an inference-time data valuation problem and introduce the Contextual Influence Value (CI value)<n>This novel metric quantifies context quality by measuring the performance degradation when removing each context from the list.
  arXiv  Detail & Related papers  (2025-09-21T07:19:09Z)
• Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios [54.58186816693791]
  environments constantly change over time and space, posing significant challenges for object detectors trained based on a closed-set assumption.<n>We propose a new mechanism, converting the fine-tuning process to a specific- parameter generation.<n>In particular, we first design a dual-path LoRA-based domain-aware adapter that disentangles features into domain-invariant and domain-specific components.
  arXiv  Detail & Related papers  (2025-06-30T17:14:12Z)
• AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption [3.805501490912696]
  Federated fine-tuning has emerged as a promising approach to adapt foundation models to downstream tasks using decentralized data.<n>We propose AFLoRA, an adaptive and lightweight federated fine-tuning framework for Large Language Models.
  arXiv  Detail & Related papers  (2025-05-30T16:35:32Z)
• Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation [52.3707788779464]
  We introduce a novel Jensen-Shannon Divergence driven method to Attribute Response to Context (ARC-JSD)<n>ARC-JSD enables efficient and accurate identification of essential context sentences without additional fine-tuning, gradient-calculation or surrogate modelling.<n> Evaluations on a wide range of RAG benchmarks, such as TyDi QA, Hotpot QA, and Musique, using instruction-tuned LLMs in different scales demonstrate superior accuracy and significant computational efficiency improvements.
  arXiv  Detail & Related papers  (2025-05-22T09:04:03Z)
• Hallucinations and Truth: A Comprehensive Accuracy Evaluation of RAG, LoRA and DoRA [0.0]
  Recent advancements in Generative AI have significantly improved the efficiency and adaptability of natural language processing (NLP) systems.<n>This paper presents a large-scale empirical evaluation of RAG, LoRA, and DoRA, with model fine-tuning and generation performance assessed on 20,000 FAQ-based queries.<n>DoRA achieves the highest accuracy (90.1%), relevance score (0.88), and lowest latency (110 ms per query), outperforming both LoRA and RAG in real-world, domain-specific generative AI applications.
  arXiv  Detail & Related papers  (2025-02-14T17:38:25Z)
• ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation [8.223134723149753]
  We present an unsupervised framework that optimize Retrieval-Augmented Generation (RAG) systems.<n>By decomposing paragraphs into sentences, we dynamically re-weighting core content while preserving contextual coherence.<n>This framework has been validated across various datasets, Large Language Models (LLMs), and retrievers.
  arXiv  Detail & Related papers  (2025-02-12T07:32:48Z)
• GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems [8.669397145785942]
  We propose Generative Error Correction via Retrieval-Augmented Generation (GEC-RAG) to improve ASR accuracy for low-resource domains, like Persian.<n>GEC-RAG retrieves lexically similar examples to the ASR transcription using the Term Frequency-Inverse Document Frequency (TF-IDF) measure.
  arXiv  Detail & Related papers  (2025-01-18T11:53:22Z)
• Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients [6.570712059945705]
  Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation.<n>Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings.<n>We introduce LoRA-A$2$, which demonstrates robustness in challenging settings with low ranks and high data heterogeneity.
  arXiv  Detail & Related papers  (2024-10-30T08:48:21Z)
• RAGGED: Towards Informed Design of Scalable and Stable RAG Systems [51.171355532527365]
  Retrieval-augmented generation (RAG) enhances language models by integrating external knowledge.<n>RAGGED is a framework for systematically evaluating RAG systems.
  arXiv  Detail & Related papers  (2024-03-14T02:26:31Z)
• Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability [73.34532767873785]
  We propose the concept of Information Density'' (ID) to indicate whether a matrix strongly belongs to certain feature spaces. We introduce the Dense Information Prompt (DIP) to enhance information density to improve generalization. DIP significantly reduces the number of tunable parameters and the requisite storage space, making it particularly advantageous in resource-constrained settings.
  arXiv  Detail & Related papers  (2023-12-17T20:42:43Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)
• Hyper-parameter Adaptation of Conformer ASR Systems for Elderly and Dysarthric Speech Recognition [64.9816313630768]
  Fine-tuning is often used to exploit the large quantities of non-aged and healthy speech pre-trained models. This paper investigates hyper- parameter adaptation for Conformer ASR systems that are pre-trained on the Librispeech corpus.
  arXiv  Detail & Related papers  (2023-06-27T07:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.