SPEAR: Code-Augmented Agentic Prompt Optimization

• URL: http://arxiv.org/abs/2605.26275v1
• Date: Mon, 25 May 2026 19:01:10 GMT
• Title: SPEAR: Code-Augmented Agentic Prompt Optimization
• Abstract summary: SPEAR (SandFrame Prompt Engineer with Active Roll-back) is a free-form agentic with four tools that decides autonomously how and when to use them.<n>The Python tool is the largest single lever on complex judge tasks.
• Score: 27.161602978517706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed pipeline. We port the code-as-action paradigm of CodeAct (Wang et al., 2024a) to APE and propose SPEAR (Sandboxed Prompt Engineer with Active Roll-back), a free-form agentic optimizer with four tools -- evaluate, python, set_prompt, finish -- that decides autonomously how and when to use them. The distinctive tool is the Python sandbox: the optimizer writes and executes arbitrary Python on the current evaluation DataFrame, performing structural error analysis (confusion matrices, error clustering, per group metrics) the agent itself authors. Two guardrails turn the long-horizon agent into a monotone-improving optimizer: auto-rollback on metric regression, and an optional guard metric floor. We evaluate on three industrial LLM-as-judge suites (13 judge tasks across recruiter-intake, conversational-memory, and query-refinement systems) plus seven BBH tasks and GSM8K. SPEAR wins every industrial task on the primary metric ($κ$ 0.857 vs 0.359 on tool-selection; F1-macro 0.815 vs 0.763 on filter-relevance; $κ$ 0.254 vs 0.218 on the hardest extraction dimension). On BBH-7 SPEAR averages 0.938 accuracy vs GEPA 0.628 and TextGrad 0.484. Ablations show the Python tool is the largest single lever on complex judge tasks ($Δ\approx +0.79κ$ on the 5-class tool-selection judge, $Δ\approx +0.35κ$ on the hardest extraction dimension when removed); its irreplaceable contribution is class-pair confusion aggregation that a long-context LLM cannot extract reliably from the raw eval DataFrame.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair [13.668549452625117]
  Kozuchi Agent is a language-agnostic open-weight repair agent and CI-operated evaluation pipeline.<n>Explicit phases, persistent state, deterministic tools, a model-independent action interface, and cross-agent test-time selection make runs auditable and repeatable.
  arXiv  Detail & Related papers  (2026-08-16T07:06:37Z)
• InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy [0.9924185669370708]
  InvestPhilBench is a dynamic benchmark spanning eight cognitive tiers.<n>The v0.6 release comprises 118 primary-source-verified investment principle cards.<n>For reproducible scoring at scale we introduce the Benchmark Automated Scoring Pipeline.
  arXiv  Detail & Related papers  (2026-06-24T15:53:20Z)
• Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents [5.689042186242701]
  Decision rules that enterprise experts apply tacitly can be systematically recovered and improved through iterative error analysis.<n>We present textbfTrace2Policy, whose core mechanism is textbfEISR.<n>Each round executes the rules on a validation set, clusters errors by root cause into MISSING, WRONG, or CONFLICT types, applies targeted patches, and commits only those that pass a regression gate.
  arXiv  Detail & Related papers  (2026-06-09T06:05:29Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection [1.2647816797166167]
  We introduce Bits-over-Random (BoR), a measure of retrieval selectivity that reveals when high success rates mask random-level performance.<n>We show that when the expected coverage ratio $left(fracK cdot barR_qNright)$ exceeds 3-5, the baseline dominates and selectivity collapses.<n>These findings suggest reporting BoR alongside traditional metrics and reconsidering depth choices when additional retrieval provides negligible selectivity gains.
  arXiv  Detail & Related papers  (2026-05-14T00:19:57Z)
• Formalize, Don't Optimize: The Heuristic Trap in LLM-Generated Combinatorial Solvers [52.23061619664667]
  Large Language Models (LLMs) struggle to solve complex problems through direct reasoning, so recent neuro-symbolic systems increasingly use them to synthesize executable solvers.<n>We introduce CP-SynC-XL, a benchmark of 100 problems (4,577 instances), and evaluate three solver--construction paradigms: native algorithmic search (Python), constraint modeling through a Python solver API (Python + OR-Tools), and declarative constraint modeling.
  arXiv  Detail & Related papers  (2026-05-12T17:15:45Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• FGDM: Reasoning Aware Multi-Agentic Framework for Software Bug Detection using Chain of Thought and Tree of Thought Prompting [0.0]
  The Flow-Graph-Driven Multi-Agent Framework (FGDM) is composed of four agents that operate in a sequential manner.<n>The framework converts the received code to a flow graph, identifies the erroneous segments, and further generates the repaired code.
  arXiv  Detail & Related papers  (2026-04-27T17:22:15Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference [12.135271159221178]
  Outcome-Aware Tool Selection (OATS) interpolates tool embeddings toward the centroid of queries where they historically succeed.<n>On MetaTool (199tools, 4,287queries), this improves NDCG@5 from 0.869 to 0.940.<n>All methods are evaluated on the same held-out 30% test split.
  arXiv  Detail & Related papers  (2026-03-13T02:13:05Z)
• PyBADS: Fast and robust black-box optimization in Python [11.4219428942199]
  PyBADS is an implementation of the Adaptive Direct Search (BADS) algorithm for fast and robust black-box optimization. It comes along with an easy-to-use Python interface for running the algorithm for running the results.
  arXiv  Detail & Related papers  (2023-06-27T15:54:44Z)
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization [63.55408755562274]
  A judge large language model, named PandaLM, is trained to distinguish the superior model given several large language models. PandaLM addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. PandaLM achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset.
  arXiv  Detail & Related papers  (2023-06-08T10:41:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.