MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization

• URL: http://arxiv.org/abs/2607.11944v1
• Date: Sat, 11 Jul 2026 10:05:58 GMT
• Title: MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization
• Abstract summary: MAGE (Memory-Augmented Goal Prompt Evolution) is a controlled analysis framework for studying component interaction in prompt optimization.<n>Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE), when multiple optimization signals operate within a closed reflective loop.
• Score: 3.7323766833175718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do different components of iterative prompt optimization interact, and what happens when they are combined? We investigate this through MAGE (Memory-Augmented Goal-directed Prompt Evolution), a controlled analysis framework for studying component interaction in prompt optimization. MAGE is not proposed as a superior optimizer in absolute terms; it integrates episodic memory, multi-objective Pareto selection, and adaptive evaluation as a platform for controlled ablation. Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE): when multiple stochastic optimization signals operate within a closed reflective loop, they interact in ways that simultaneously improve performance and amplify variance, behavior that cannot be predicted by analyzing components in isolation. Three main findings emerge. First, failure-grounded reflection is essential: methods relying only on scores (OPRO) or abstract critique (Self-Refine) fail to improve prompts. Second, MAGE achieves 46.4% versus GEPA's 34.0% on GSM8K-Hard (+12.4%, P(MAGE>GEPA)=0.998, 5 seeds on gpt-4o-mini), with comparable variance (7.3% vs. 7.0%). Third, increasing candidate diversity reveals the clearest POCE signal: expanding the candidate pool from n=3 to n=5 improves mean accuracy by +21.6% while increasing variance by 3.7x. We further validate on Llama 3.1 8B and show POCE is headroom-dependent: when the base model already achieves high accuracy, variance amplification disappears. Finally, in low-data regimes (Ntrain=30), well-designed fixed prompts outperform all reflective optimizers, indicating that scaffold choice dominates optimizer choice. Our results suggest prompt optimization systems behave as coupled stochastic processes and should be evaluated in terms of both performance and stability, not just peak accuracy.

Related papers

• ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize [21.715943293929296]
  ESPO decomposes prompt optimization into three phases: Diagnose clusters all training errors in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability.<n>On seven public NLP benchmarks, ESPO improves average accuracy by $+$3.76 pp over the state-of-the-art (74.67% vs 70.91% for GEPA)
  arXiv  Detail & Related papers  (2026-09-03T17:59:37Z)
• On-Policy Self-Distillation in Diffusion Models [89.32656931795293]
  Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
  arXiv  Detail & Related papers  (2026-08-25T14:55:24Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• Spokes: Optimizing for Diverse Pretraining Data Selection [82.66872118512403]
  We introduce a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent.<n>Our method produces subsets substantially more diverse than those obtained via random sampling.<n>We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2026-06-13T09:28:46Z)
• ContraPrompt: Contrastive Prompt Optimization via Dyadic Reasoning Trace Analysis [0.6372261626436676]
  We introduce ContraPrompt, built on the observation that when a model fails but succeeds on a retry with feedback, the difference constitutes an optimization signal.<n>Unlike prior contrastive methods, we compare complete intermediate reasoning processes.<n>ContraPrompt beats GEPA on 11, ties on 41, and loses on 1 at equal budget.
  arXiv  Detail & Related papers  (2026-04-20T08:17:15Z)
• Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees [28.54776477263591]
  We present Prompt-Aware Online Evaluation Scheduling (POES)<n>POES integrates an IRT-based discrimination utility, a facility-location coverage term, and switching-cost-aware warm-start swaps into a unified objective.<n>POES achieves the highest overall average accuracy (6.2 percent improvement over the best baseline) with negligible token overhead.
  arXiv  Detail & Related papers  (2026-04-13T11:31:04Z)
• Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE [7.609008983716641]
  We conduct experiments on multimodal sequential recommendation to compare common negative-selection strategies and their interaction with DPO training.<n>Our central finding is that a simple modification, replacing deterministic hard negatives with sampling from a dynamic top-K candidate pool, consistently improves ranking performance.
  arXiv  Detail & Related papers  (2026-03-31T04:49:32Z)
• CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning [17.364321308755667]
  CoT2-Meta is a training-free metacognitive reasoning framework.<n>It combines object-level chain-of-thought generation with meta-level control over partial reasoning trajectories.<n>It consistently outperforms strong single-path, sampling-based, and search-based baselines.
  arXiv  Detail & Related papers  (2026-03-30T07:59:47Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment [57.03947082589616]
  Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. We study this and find that preference data gives a better learning signal when the underlying responses are contrastive. We introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%.
  arXiv  Detail & Related papers  (2024-08-12T16:24:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.