CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG

• URL: http://arxiv.org/abs/2605.08333v1
• Date: Fri, 08 May 2026 17:58:11 GMT
• Title: CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG
• Abstract summary: We propose CDS4RAG, a framework to optimize the Retrieval-Augmented Generation using given queries.<n>We show that CDS4RAG significantly boosts the vanilla algorithms in 21/24 cases while significantly outperforming state-of-the-art algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.
• Score: 4.208654120634349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) is sensitive to the vast hyperparameters of the retriever and generator, yet optimizing them using given queries is a challenging task due to the complex interactions and expensive evaluation costs. Existing algorithms are ineffective and slow in convergence, since they often treat RAG as a monolithic black box or only optimize partial hyperparameters. In this paper, we propose CDS4RAG, a framework that optimizes the full RAG hyperparameters using given queries via a new cyclic dual-sequential formulation. CDS4RAG is special in the sense that it distinguishes the hyperparameters of the retriever and generator, cyclically optimizing them in turn. Such a paradigm allows us to design fine-grained within-cycle budget provision and expedite the optimization via cross-cycle seeding when optimizing the generator. CDS4RAG is also an algorithm-agnostic framework that can be paired with diverse general algorithms. Through experiments on four common benchmarks and two backbone LLMs, we reveal that CDS4RAG considerably boosts the vanilla algorithms in 21/24 cases while significantly outperforming state-of-the-art algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.

Related papers

• optimize_anything: A Universal API for Optimizing any Text Parameter [98.42497715725356]
  We show a single AI-based optimization system-supporting single-task search, multi-task search with cross-problem transfer, and generalization to unseen inputs.<n>We show for the first time that text optimization with LLM-based search is a general-purpose problem-solving paradigm.
  arXiv  Detail & Related papers  (2026-05-19T10:18:12Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning [23.59600455731982]
  We propose a joint optimization framework that simultaneously optimize the bit-width and LoRA rank configuration for each layer during the mixed quantized fine-tuning process.<n>Experiments show that AutoQRA achieves performance close to full-precision fine-tuning with a memory footprint comparable to uniform 4-bit methods.
  arXiv  Detail & Related papers  (2026-02-25T07:18:08Z)
• A Dynamic Retrieval-Augmented Generation System with Selective Memory and Remembrance [0.0]
  We introduce emph Adaptive RAG Memory (ARM), a retrieval-augmented generation (RAG) framework that replaces a static vector index with a emphdynamic memory substrate.<n>ARM reaches near state-of-the-art performance on a lightweight retrieval benchmark.<n>ARM yields competitive accuracy, self-regularizing memory growth, and interpretable retention dynamics without retraining the generatorcolorblack and provides practical trade-off between quality, latency and memory efficiency for production and research RAG system.
  arXiv  Detail & Related papers  (2026-01-04T21:51:41Z)
• Genetic Algorithms For Parameter Optimization for Disparity Map Generation of Radiata Pine Branch Images [5.266753902938501]
  We propose a Genetic Algorithm (GA) based parameter optimization framework for UAV applications.<n>Our contributions include: (1) a novel GA-based parameter optimization framework that eliminates manual tuning; (2) a comprehensive evaluation methodology using multiple image quality metrics; and (3) a practical solution for resource-constrained UAV systems.<n> Experimental results demonstrate that our GA-optimized approach reduces Mean Squared Error by 42.86% while increasing Peak Signal-to-Noise Ratio and Structural Similarity by 8.47% and 28.52%, respectively, compared with baseline configurations.
  arXiv  Detail & Related papers  (2025-12-05T04:00:18Z)
• Causal-Guided Dimension Reduction for Efficient Pareto Optimization [2.9013001432962255]
  CaDRO builds a causal map through a hybrid observational-interventional process, ranking parameters by their causal effect on the objectives.<n>Low-impact parameters are fixed to values from high-quality solutions, while critical drivers remain active in the search.<n>Across amplifiers, regulators, and RF circuits, CaDRO converges up to 10$times$ faster than NSGA-II.
  arXiv  Detail & Related papers  (2025-10-11T00:41:04Z)
• An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation [6.98773220458697]
  We present a comprehensive study involving 5 HPO algorithms over 5 datasets from diverse domains.<n>Our study explores the largest HPO search space considered to date, with three evaluation metrics as optimization targets.<n>Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets.
  arXiv  Detail & Related papers  (2025-05-06T11:47:52Z)
• Parameter Optimization with Conscious Allocation (POCA) [4.478575931884855]
  Hyperband-based approaches to machine learning are among the most effective. We present. the new. Optimization with Conscious Allocation (POCA), a hyperband-based algorithm that adaptively allocates the inputted. budget to the hyperparameter configurations it generates. POCA finds strong configurations faster in both settings.
  arXiv  Detail & Related papers  (2023-12-29T00:13:55Z)
• Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits [55.03293214439741]
  In contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience. We propose the first online continuous hyperparameter tuning framework for contextual bandits. We show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-02-18T23:31:20Z)
• Provably Faster Algorithms for Bilevel Optimization [54.83583213812667]
  Bilevel optimization has been widely applied in many important machine learning applications. We propose two new algorithms for bilevel optimization. We show that both algorithms achieve the complexity of $mathcalO(epsilon-1.5)$, which outperforms all existing algorithms by the order of magnitude.
  arXiv  Detail & Related papers  (2021-06-08T21:05:30Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.