RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

• URL: http://arxiv.org/abs/2605.09730v2
• Date: Wed, 13 May 2026 21:14:46 GMT
• Title: RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement
• Abstract summary: Iterative self-refinement is a popular inference-time reliability technique.<n>But its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal.<n>We introduce a training-free pre-execution reliability layer that generates task- and registry-specific rubrics.
• Score: 0.36165327398913766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly ($0.75$ vs. $0.65$ baseline). The dominant failures are inter-tool contract violations - wrong output shape, incorrect tool routing, broken argument provenance - that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free pre-execution reliability layer that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. With zero execution attempts, RubricRefine reaches $0.86$ on M3ToolEval averaged across seven models-improving over prior inference-time baselines on every model tested on this benchmark, at $2.6X$ lower latency than the strongest non-iterative alternative - and remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. A rubric-category ablation and calibration analysis further characterize when and why the method works.

Related papers

• ExecuCritic: Calibrated Critic Shaping for Code Generation with Verifiable Rewards [2.724374986592201]
  Execution feedback is a useful supervision signal for code models because unit tests are objective and directly measure program correctness.<n>We propose ExecuCritic, a joint training framework in which a coder and a critic are updated on the same execution rollouts.
  arXiv  Detail & Related papers  (2026-09-15T03:58:21Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback [13.976149104483449]
  PAIR-Bench is a progressive benchmark for evaluating large language models (LLMs)<n>It transforms an incorrect or incomplete program into a more correct one through feedback-guided refinement.<n>It measures whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires.
  arXiv  Detail & Related papers  (2026-07-01T18:20:27Z)
• To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair [18.653708704742353]
  Executions can be time-consuming and expensive, yet their impact on program repair agents remains underexplored.<n>Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task.<n>Current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit.
  arXiv  Detail & Related papers  (2026-06-25T12:49:59Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics [15.131840310228712]
  RIFT is a taxonomy for systematically characterizing failure modes in rubric composition and design.<n>RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures.
  arXiv  Detail & Related papers  (2026-04-01T20:34:43Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• $V_1$: Unifying Generation and Self-Verification for Parallel Reasoners [69.66089681814013]
  $V_$ is a framework that unifies generation and verification through efficient pairwise ranking.<n>$V_$-Infer improves Pass@1 by up to $10%$ over pointwise verification.<n>$V_$-PairRL achieves $7$--$9%$ test-time scaling gains over standard RL and pointwise joint training.
  arXiv  Detail & Related papers  (2026-03-04T17:22:16Z)
• Trajectory Guard -- A Lightweight, Sequence-Aware Model for Real-Time Anomaly Detection in Agentic AI [0.0]
  Trajectory Guard is a Siamese Recurrent Autoencoder with a hybrid loss function that jointly learns task-trajectory alignment via contrastive learning and sequential validity via reconstruction.<n>At 32 ms latency, our approach runs 17$-27times$ faster than LLM Judge baselines, enabling real-time safety verification in production deployments.
  arXiv  Detail & Related papers  (2026-01-02T00:27:11Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• GRETEL: A Goal-driven Retrieval and Execution-based Trial Framework for LLM Tool Selection Enhancing [6.790345342526922]
  GRETEL implements an agentic workflow that processes semantically retrieved candidates through sandboxed plan-execute-evaluate cycles.<n>Our comprehensive evaluation on the ToolBench benchmark demonstrates substantial improvements across all metrics.
  arXiv  Detail & Related papers  (2025-10-10T00:12:51Z)
• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions [10.598440138966028]
  Current self-reflection practices rely on prompts or one-way reasoning.<n>We propose structured reflection, which turns the path from error to repair into an explicit, controllable, and trainable action.<n>Experiments on BFCL v3 and Tool-Reflection-Bench show large gains in multi-turn tool-call success and error recovery, and a reduction of redundant calls.
  arXiv  Detail & Related papers  (2025-09-23T09:35:49Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.