No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization

• URL: http://arxiv.org/abs/2509.23387v1
• Date: Sat, 27 Sep 2025 16:13:26 GMT
• Title: No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization
• Authors: Wenhang Shi, Yiren Chen, Shuqing Bian, Xinyi Zhang, Kai Tang, Pengfei Hu, Zhe Zhao, Wei Lu, Xiaoyong Du,
• Abstract summary: GRACE is a framework that integrates two synergistic strategies: Gated Refinement and Adaptive Compression.<n>GRACE achieves significant average relative performance improvements of 4.7%, 4.4% and 2.7% over state-of-the-art methods, respectively.
• Score: 15.610820088794796
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt engineering is crucial for leveraging the full potential of large language models (LLMs). While automatic prompt optimization offers a scalable alternative to costly manual design, generating effective prompts remains challenging. Existing methods often struggle to stably generate improved prompts, leading to low efficiency, and overlook that prompt optimization easily gets trapped in local optima. Addressing this, we propose GRACE, a framework that integrates two synergistic strategies: Gated Refinement and Adaptive Compression, achieving Efficient prompt optimization. The gated refinement strategy introduces a feedback regulation gate and an update rejection gate, which refine update signals to produce stable and effective prompt improvements. When optimization stagnates, the adaptive compression strategy distills the prompt's core concepts, restructuring the optimization trace and opening new paths. By strategically introducing information loss through refinement and compression, GRACE delivers substantial gains in performance and efficiency. In extensive experiments on 11 tasks across three practical domains, including BIG-Bench Hard (BBH), domain-specific, and general NLP tasks, GRACE achieves significant average relative performance improvements of 4.7%, 4.4% and 2.7% over state-of-the-art methods, respectively. Further analysis shows that GRACE achieves these gains using only 25% of the prompt generation budget required by prior methods, highlighting its high optimization efficiency and low computational overhead. Our code is available at https://github.com/Eric8932/GRACE.

Related papers

• Learning from Prompt itself: the Hierarchical Attribution Prompt Optimization [13.8868879878572]
  A structured optimization approach requires automated or semi-automated procedures to develop improved prompts.<n>Current prompt optimization methods often induce prompt drift, where new prompts fix prior failures but impair performance on previously successful tasks.<n>This study proposes the Hierarchical Prompt Optimization framework, which introduces three innovations: (1) a dynamic attribution mechanism targeting error patterns in training data and prompting history, (2) semantic-unit optimization for editing functional prompt segments, and (3) multimodal-friendly progression supporting both end-to-end LLM and LLM-MLLM.
  arXiv  Detail & Related papers  (2026-01-06T03:34:17Z)
• Optimizing Optimizers for Fast Gradient-Based Learning [53.81268610971847]
  We lay the theoretical foundation for automating design in gradient-based learning.<n>By treating gradient loss signals as a function that translates to parameter motions, the problem reduces to a family of convex optimization problems.
  arXiv  Detail & Related papers  (2025-12-06T09:50:41Z)
• MetaML-Pro: Cross-Stage Design Flow Automation for Efficient Deep Learning Acceleration [8.43012094714496]
  This paper presents a unified framework for codifying and automating optimization strategies to deploy deep neural networks (DNNs) on resource-constrained hardware.<n>Our novel approach addresses two key issues: (i)encoding custom optimization strategies and (ii)enabling cross-stage optimization search.
  arXiv  Detail & Related papers  (2025-02-09T11:02:06Z)
• StraGo: Harnessing Strategic Guidance for Prompt Optimization [35.96577924228001]
  StraGo is a novel approach designed to mitigate prompt drifting by leveraging insights from both successful and failed cases. It employs a how-to-do methodology, integrating in-context learning to formulate specific, actionable strategies. Experiments conducted across a range of tasks, including reasoning, natural language understanding, domain-specific knowledge, and industrial applications, demonstrate StraGo's superior performance.
  arXiv  Detail & Related papers  (2024-10-11T07:55:42Z)
• Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System [75.25394449773052]
  Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving.<n>Yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods.<n>We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness.
  arXiv  Detail & Related papers  (2024-10-10T17:00:06Z)
• Localized Zeroth-Order Prompt Optimization [54.964765668688806]
  We propose a novel algorithm, namely localized zeroth-order prompt optimization (ZOPO) ZOPO incorporates a Neural Tangent Kernel-based derived Gaussian process into standard zeroth-order optimization for an efficient search of well-performing local optima in prompt optimization. Remarkably, ZOPO outperforms existing baselines in terms of both the optimization performance and the query efficiency.
  arXiv  Detail & Related papers  (2024-03-05T14:18:15Z)
• Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers [108.72225067368592]
  We propose a novel perspective to investigate the design of large language models (LLMs)-based prompts.<n>We identify two pivotal factors in model parameter learning: update direction and update method.<n>We develop a capable Gradient-inspired Prompt-based GPO.
  arXiv  Detail & Related papers  (2024-02-27T15:05:32Z)
• Large Language Models as Optimizers [106.52386531624532]
  We propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as prompts. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values. We demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
  arXiv  Detail & Related papers  (2023-09-07T00:07:15Z)
• B2Opt: Learning to Optimize Black-box Optimization with Little Budget [15.95406229086798]
  This paper designs a powerful optimization framework to automatically learn the optimization strategies from the target or cheap surrogate task without human intervention. Deep neural network framework called B2Opt has a stronger representation of optimization strategies based on survival of the fittest. Compared to the state-of-the-art BBO baselines, B2Opt can achieve multiple orders of magnitude performance improvement with less function evaluation cost.
  arXiv  Detail & Related papers  (2023-04-24T01:48:01Z)
• DECN: Evolution Inspired Deep Convolution Network for Black-box Optimization [9.878660285945728]
  This paper introduces the concept of Automated EA: Automated EA exploits structure in the problem of interest to automatically generate update rules.<n>We design a deep evolutionary convolution network (DECN) to realize the move from hand-designed EAs to automated EAs without manual interventions.
  arXiv  Detail & Related papers  (2023-04-19T12:14:01Z)
• A Simple Baseline for StyleGAN Inversion [133.5868210969111]
  StyleGAN inversion plays an essential role in enabling the pretrained StyleGAN to be used for real facial image editing tasks. Existing optimization-based methods can produce high quality results, but the optimization often takes a long time. We present a new feed-forward network for StyleGAN inversion, with significant improvement in terms of efficiency and quality.
  arXiv  Detail & Related papers  (2021-04-15T17:59:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.