Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds

• URL: http://arxiv.org/abs/2606.29654v1
• Date: Sun, 28 Jun 2026 23:46:56 GMT
• Title: Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
• Abstract summary: We formulate this as budgeted act-or-defer decision making.<n>System maps the debate prefix to a low-dimensional state.<n>It acts only when the bound exceeds a user-specified reliability threshold.
• Score: 8.682220907885714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review. We formulate this as budgeted act-or-defer decision making. At each round, the system maps the debate prefix to a low-dimensional state, computes a $k$-nearest-neighbor lower confidence bound on state-conditional correctness using calibration data, and acts only when the bound exceeds a user-specified reliability threshold. The certificate controls wrong actions through the decomposition $β= δ+ α+ \varepsilon_{\mathrm{act}}$, separating calibration failure, residual action risk, and representation gap. The guarantee is conditional, not distribution-free: it relies on a valid local bias envelope and an action-region representation-gap bound, and each assumption is paired with falsification-style diagnostics. Because the same absolute wrong-action budget has different meanings across tasks of different difficulty, we set budgets relative to each task's final-round error using training data only, and evaluate safety by normalized budget usage $\mathrm{WA}/β$. On six benchmarks against nine baselines, the method uses 9--12% of the pre-declared budget on activated datasets, reaching up to 84% automation and 96% acted-on accuracy; on stress-test datasets, it defers rather than forcing unreliable automation. Rather than relying on per-task post-hoc threshold search, the method prospectively converts a user-declared wrong-action budget into an auditable act-or-defer operating point before deployment, under explicitly stated assumptions.

Related papers

• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Lottery Tickets Are Not Deployment Tickets [4.230349220081444]
  We study whether an accuracy-matched lottery ticket or another sparse challenger can replace an incumbent dense model without reconfiguring downstream decision logic.<n>Across experiments, sparse candidates repeatedly recover dense-reference accuracy yet remain behaviorally different.<n>In small-gap settings with fixed-threshold policy diagnostics, lottery-ticket replacement changes 7% to 10% of accept-review decisions.
  arXiv  Detail & Related papers  (2026-07-29T15:25:19Z)
• A Stackelberg Framework for Resource-Aware LLM Agents: Learning, Repair, and Conditional Guarantees [1.9381445674403615]
  Large language model (LLM) agents increasingly operate as multi-turn systems must allocate context, prompt verbosity, and tool access under finite computational budgets.<n>We formulate resource governance as a contextual Stackelberg game: a controller commits to a quality target and a cost incentive, while an executor responds with resource actions over context, prompting, and tool usage.<n>We learn a conditional response model, optimize a leader policy against that model, and repair the resulting policy using real-API calibration and projection.
  arXiv  Detail & Related papers  (2026-06-22T08:38:22Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Instance-Optimal Estimation with Multiple LLM Judges on a Budget [84.31744861038106]
  We formalize this question as *budgeted heteroskedastic multi-judge estimation*.<n>Given $K$ prompt-response pairs, $J$ judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an $ell_p$-error.<n>We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget.
  arXiv  Detail & Related papers  (2026-05-22T08:26:08Z)
• Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning [8.024041325202612]
  Chain-of-thought (CoT) reasoning with self-consistency improves performance by aggregating multiple sampled reasoning paths.<n>We introduce a conformal procedure for CoT reasoning that directly addresses aggregation uncertainty.<n>Our approach replaces majority voting with weighted score aggregation over reasoning paths and calibrates an abstention rule using conformal risk control.
  arXiv  Detail & Related papers  (2026-05-13T20:33:59Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization [18.737087162461563]
  Test-time compute scaling has become a powerful lever for improving large language model performance.<n>Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore.<n>We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline.
  arXiv  Detail & Related papers  (2026-04-16T10:39:22Z)
• Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades [29.43134687557432]
  Existing cascades delegate based on probe uncertainty, but uncertainty is a poor proxy for delegation benefit.<n>We introduce Calibrate-Then-Delegate (CTD), a model-cascade approach that provides probabilistic guarantees on the computation cost.<n>CTD consistently outperforms uncertainty-based delegation at every budget level.
  arXiv  Detail & Related papers  (2026-04-15T11:05:59Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Conformal Tradeoffs: Guarantees Beyond Coverage [0.02648566468224904]
  Deployed conformal predictors are long-lived decision infrastructure operating over finite operational windows.<n> Marginal coverage does not determine deployment-facing quantities.<n>We provide a framework for operational certification and planning beyond coverage.
  arXiv  Detail & Related papers  (2026-02-20T07:58:25Z)
• Conformal Thinking: Risk Control for Reasoning on a Compute Budget [60.65072883773352]
  Reasoning Large Language Models (LLMs) enable test-time scaling, with dataset-level accuracy improving as the token budget increases.<n>We re-frame the budget setting problem as risk control, limiting the error rate while minimizing compute.<n>Our framework introduces an upper threshold that stops reasoning when the model is confident and a novel lower threshold that preemptively stops unsolvable instances.
  arXiv  Detail & Related papers  (2026-02-03T18:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.