Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario

• URL: http://arxiv.org/abs/2406.12429v3
• Date: Mon, 30 Sep 2024 06:04:05 GMT
• Title: Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario
• Authors: Feiteng Mu, Yong Jiang, Liwen Zhang, Chu Liu, Wenjie Li, Pengjun Xie, Fei Huang,
• Abstract summary: Current research on tool learning primarily focuses on selecting the most effective tool from a wide array of options, often overlooking cost-effectiveness. In this paper, we address the selection of homogeneous tools by predicting both their performance and the associated cost required to accomplish a given task.
• Score: 62.615210194004106
• License:
• Abstract: Current research on tool learning primarily focuses on selecting the most effective tool from a wide array of options, often overlooking cost-effectiveness, a crucial factor in human problem-solving. In this paper, we address the selection of homogeneous tools by predicting both their performance and the associated cost required to accomplish a given task. We then assign queries to the optimal tools in a cost-effective manner. Our experimental results demonstrate that our method achieves higher performance at a lower cost compared to strong baseline approaches.

Related papers

• Compute-Constrained Data Selection [77.06528009072967]
  We formalize the problem of data selection with a cost-aware utility function, and model the problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute.
  arXiv  Detail & Related papers  (2024-10-21T17:11:21Z)
• Sample-Optimal Large-Scale Optimal Subset Selection [0.9558392439655016]
  We design a top-$m$ greedy selection mechanism that keeps sampling the current top $m$ alternatives with top $m$ running sample means. We prove that the EFG-$m$ procedure is both sample optimal and consistent in terms of the probability of good selection. Surprisingly, we also demonstrate that the EFG-$m$ procedure enables to achieve an indifference-based ranking within the selected subset of alternatives at no extra cost.
  arXiv  Detail & Related papers  (2024-08-18T16:44:41Z)
• Training Greedy Policy for Proposal Batch Selection in Expensive Multi-Objective Combinatorial Optimization [52.80408805368928]
  We introduce a novel greedy-style subset selection algorithm for batch acquisition. Our experiments on the red fluorescent proteins show that our proposed method achieves the baseline performance in 1.69x fewer queries.
  arXiv  Detail & Related papers  (2024-06-21T05:57:08Z)
• Budget-Constrained Tool Learning with Planning [40.52601704021853]
  This paper proposes a novel method for budget-constrained tool learning. Our approach involves creating a preferable plan under the budget constraint before utilizing the tools.
  arXiv  Detail & Related papers  (2024-02-25T02:46:33Z)
• Efficient Prompt Optimization Through the Lens of Best Arm Identification [50.56113809171805]
  This work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. It is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB)
  arXiv  Detail & Related papers  (2024-02-15T05:31:13Z)
• Experience-Based Evolutionary Algorithms for Expensive Optimization [8.466374531816427]
  We argue that hard optimization problems could be tackled efficiently by making better use of experiences gained in related problems. We propose an experience-based surrogate-assisted evolutionary algorithm (SAEA) framework to enhance the optimization efficiency of expensive problems.
  arXiv  Detail & Related papers  (2023-04-09T05:47:14Z)
• Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach [31.918476422203412]
  This paper proposes a novel approach referred to as Budget-Aware Planning for Iterative learners (BAPI) to solve HPO problems under a constrained cost budget. Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most of the cases.
  arXiv  Detail & Related papers  (2022-06-25T18:44:06Z)
• Compactness Score: A Fast Filter Method for Unsupervised Feature Selection [66.84571085643928]
  We propose a fast unsupervised feature selection method, named as, Compactness Score (CSUFS) to select desired features. Our proposed algorithm seems to be more accurate and efficient compared with existing algorithms.
  arXiv  Detail & Related papers  (2022-01-31T13:01:37Z)
• Reinforcement Learning with Efficient Active Feature Acquisition [59.91808801541007]
  In real-life, information acquisition might correspond to performing a medical test on a patient. We propose a model-based reinforcement learning framework that learns an active feature acquisition policy. Key to the success is a novel sequential variational auto-encoder that learns high-quality representations from partially observed states.
  arXiv  Detail & Related papers  (2020-11-02T08:46:27Z)
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers [29.624308090226375]
  In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed anecdotes. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learnings. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods.
  arXiv  Detail & Related papers  (2020-07-03T08:19:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.