Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning

• URL: http://arxiv.org/abs/2607.22621v1
• Date: Wed, 17 Jun 2026 02:43:58 GMT
• Title: Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning
• Abstract summary: OPTI-Q implements a plan-before-execute paradigm for multi-LLM orchestration.<n> OPTI-Q searches for plans that optimize answer quality (QoA) while trading off financial cost, latency, and energy.<n>On MMLU-Pro and SimpleQA under user-specified budgets, OPTI-Q improves average QoA by 58% and 41% over baselines at comparable cost.
• Score: 0.8620335948752809
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models (LLMs) enable strong question answering (QA), budgeted deployment is complicated by nondeterminism and heterogeneous resource profiles (cost, latency, and energy). We present OPTI-Q, a database-inspired, cost-based optimizer that implements a plan-before-execute paradigm for multi-LLM orchestration. OPTI-Q models LLM invocations as physical operators in an execution DAG and, for each question, searches for plans that optimize answer quality (QoA) while trading off financial cost, latency, and energy under user-specified resource constraints. Plans can include sequential operators that pass intermediate answers as context and parallel/blend operators that run models concurrently and merge their outputs. To search this space without executing each candidate plan, OPTI-Q uses PERFDB, a statistics catalog populated and refreshed from benchmarks and execution traces, to estimate the QoA and resource costs of both individual operators and composed subplans. Using these estimates, OPTI-Q performs Pareto-frontier search and selects a final plan based on user preferences. On MMLU-Pro and SimpleQA under user-specified budgets, OPTI-Q improves average QoA by ~58% and ~41% over baselines at comparable cost, demonstrating that database-style planning yields better quality-resource trade-offs for multi-LLM QA.

Related papers

• Multiple-Prediction-Powered Inference [57.30769275944024]
  This work provides theoretical guarantees about the minimax optimality, finite-sample performance, and normality of the MultiPPI estimator.<n>We show that MultiPPI consistently achieves lower estimation error than existing baselines.
  arXiv  Detail & Related papers  (2026-03-28T21:24:00Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data [57.996437077411315]
  We study the reasoning behavior of large language models (LLMs) under limited computation budgets.<n>We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase.<n> Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models.
  arXiv  Detail & Related papers  (2026-01-16T07:09:30Z)
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees [17.478510146434218]
  Open-weight large language model (LLM) zoos provide access to numerous high-quality models.<n>Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities.<n>We introduce MESS+, an optimization algorithm for cost-optimal LLM request routing.
  arXiv  Detail & Related papers  (2025-05-26T13:11:08Z)
• BQSched: A Non-intrusive Scheduler for Batch Concurrent Queries via Reinforcement Learning [7.738546538164454]
  A key issue in minimizing the overall makespan of data pipelines is the efficient scheduling of concurrent queries.<n>To our knowledge, BQSched is the first non-intrusive batch query scheduler via reinforcement learning.<n>Extensive experiments show that BQSched can significantly improve the efficiency and stability of batch query scheduling.
  arXiv  Detail & Related papers  (2025-04-27T07:49:01Z)
• OmniRouter: Budget and Performance Controllable Multi-LLM Routing [31.60019342381251]
  Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency.<n>We introduce Omni, a controllable routing framework for multi-LLM serving.<n>Experiments show that Omni achieves up to 6.30% improvement in response accuracy while simultaneously reducing computational costs by at least 10.15%.
  arXiv  Detail & Related papers  (2025-02-27T22:35:31Z)
• Dynamic LLM Routing and Selection based on User Preferences: Balancing Performance, Cost, and Ethics [0.6999740786886538]
  We introduce OptiRoute, an advanced model routing engine designed to dynamically select and route tasks to the optimal large language model (LLMs)<n>OptiRoute captures both functional (e.g., accuracy, speed, cost) and non-functional (e.g., helpfulness, harmlessness, honesty) criteria to efficiently match tasks with the best-fit models.<n>This makes it ideal for real-time applications in cloud-based ML platforms, personalized AI services, and regulated industries.
  arXiv  Detail & Related papers  (2025-02-23T19:23:22Z)
• PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving [89.60370366013142]
  We propose PlanGEN, a model-agnostic and easily scalable agent framework with three key components: constraint, verification, and selection agents.<n>Specifically, our approach proposes constraint-guided iterative verification to enhance performance of inference-time algorithms.
  arXiv  Detail & Related papers  (2025-02-22T06:21:56Z)
• MixLLM: Dynamic Routing in Mixed Large Language Models [57.309520357563215]
  Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency.<n>We develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment.
  arXiv  Detail & Related papers  (2025-02-09T02:26:15Z)
• Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering [16.790216473975146]
  Complex table question answering (TQA) aims to answer questions that require complex reasoning, such as multi-step or multi-category reasoning.<n>Previous approaches demonstrated notable performance by leveraging either closed-source large language models (LLMs) or fine-tuned open-weight LLMs.<n>We propose Multi-Agent Collaboration with Tool use (MACT), a framework that requires neither closed-source models nor fine-tuning.
  arXiv  Detail & Related papers  (2024-12-28T13:13:33Z)
• Budget-aware Query Tuning: An AutoML Perspective [14.561951257365953]
  Modern database systems rely on cost-based querys to come up with good execution plans for input queries. We show that by varying the costunit values one can obtain query plans that significantly outperform the default query plans.
  arXiv  Detail & Related papers  (2024-03-29T20:19:36Z)
• Efficient Prompt Optimization Through the Lens of Best Arm Identification [50.56113809171805]
  This work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. It is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB)
  arXiv  Detail & Related papers  (2024-02-15T05:31:13Z)
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration [55.35849138235116]
  We propose automatically selecting a team of agents from candidates to collaborate in a dynamic communication structure toward different tasks and domains. Specifically, we build a framework named Dynamic LLM-Powered Agent Network ($textDyLAN$) for LLM-powered agent collaboration. We demonstrate that DyLAN outperforms strong baselines in code generation, decision-making, general reasoning, and arithmetic reasoning tasks with moderate computational cost.
  arXiv  Detail & Related papers  (2023-10-03T16:05:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.