ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize

• URL: http://arxiv.org/abs/2609.04197v1
• Date: Thu, 03 Sep 2026 17:59:37 GMT
• Title: ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize
• Abstract summary: ESPO decomposes prompt optimization into three phases: Diagnose clusters all training errors in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability.<n>On seven public NLP benchmarks, ESPO improves average accuracy by $+$3.76 pp over the state-of-the-art (74.67% vs 70.91% for GEPA)
• Score: 21.715943293929296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evolutionary prompt optimizers such as GEPA suffer from prompt bloat: each iteration appends rules and caveats, producing prompts up to 3$\times$ longer yet no more accurate. We trace this to three deficiencies - incomplete error observation, limited search diversity, and unreliable selection - and propose ESPO (Error-Structured Prompt Optimization), which decomposes prompt optimization into three phases: Diagnose clusters all training errors into structural patterns in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability selection. On seven public NLP benchmarks - Tweet, MMLU, GSM8K, HotpotQA, ScoNe, HoVer, and PUPA - ESPO improves average accuracy by $+$3.76 pp over the state-of-the-art (74.67% vs 70.91% for GEPA), matching or exceeding GEPA on every dataset while producing prompts 47% shorter (1,004 vs 1,878 chars) and faster at inference. Cross-model experiments across four additional student models (Gemma 3 12B, Mistral 14B, Qwen3 32B, Claude Haiku 4.5) show ESPO yields the best average accuracy on every model tested, with the largest gap on Qwen3 GSM8K (15.00% $\to$ 91.40%). A generalization bound (Appendix) grounds each phase in a corresponding term of the test-time gap, and the ablation confirms a key prediction: adding diversity without bootstrap selection actually hurts performance ($-$1.20%).

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization [3.7323766833175718]
  MAGE (Memory-Augmented Goal Prompt Evolution) is a controlled analysis framework for studying component interaction in prompt optimization.<n>Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE), when multiple optimization signals operate within a closed reflective loop.
  arXiv  Detail & Related papers  (2026-07-11T10:05:58Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Parameter-Efficient Neuroevolution for Diverse LLM Generation: Quality-Diversity Optimization via Prompt Embedding Evolution [13.891522069967507]
  Large Language Models exhibit mode collapse, producing homogeneous outputs that fail to explore valid solution spaces.<n>We present QD-LLM, a framework for parameter-efficient neuroevolution that evolves prompt embeddings.<n>On HumanEval (164 problems), QD-LLM achieves 46.4% higher coverage and 41.4% higher QD-Score than QDAIF.
  arXiv  Detail & Related papers  (2026-05-10T22:00:15Z)
• Adaptive Consensus in LLM Ensembles via Sequential Evidence Accumulation: Automatic Budget Identification and Calibrated Commit Signals [0.0]
  DASE produces a commit-type routing partition that generalises across benchmarks.<n> Adaptive stopping, not injection bandwidth, drives accuracy.<n>Injection-based methods exhibit an inverted-U accuracy-vs-inference trajectory.
  arXiv  Detail & Related papers  (2026-05-05T19:24:10Z)
• When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models [2.064923532131528]
  Deployed language models must produce outputs that are both correct and format-compliant.<n>We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH.<n>We develop AloLab, an iterative system-prompt requiring only black-box API access to the target model.
  arXiv  Detail & Related papers  (2026-05-04T09:07:44Z)
• ContraPrompt: Contrastive Prompt Optimization via Dyadic Reasoning Trace Analysis [0.6372261626436676]
  We introduce ContraPrompt, built on the observation that when a model fails but succeeds on a retry with feedback, the difference constitutes an optimization signal.<n>Unlike prior contrastive methods, we compare complete intermediate reasoning processes.<n>ContraPrompt beats GEPA on 11, ties on 41, and loses on 1 at equal budget.
  arXiv  Detail & Related papers  (2026-04-20T08:17:15Z)
• S*: Test Time Scaling for Code Generation [55.11863577956177]
  We propose S*, the first hybrid test-time scaling framework for code generation.<n>S* substantially improves the coverage and selection accuracy of generated code.
  arXiv  Detail & Related papers  (2025-02-20T09:18:53Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.