Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection

• URL: http://arxiv.org/abs/2504.01931v2
• Date: Sat, 05 Apr 2025 15:45:13 GMT
• Title: Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection
• Authors: Souradip Chakraborty, Mohammadreza Pourreza, Ruoxi Sun, Yiwen Song, Nino Scherrer, Furong Huang, Amrit Singh Bedi, Ahmad Beirami, Jindong Gu, Hamid Palangi, Tomas Pfister,
• Abstract summary: Inference-time methods such as Best-of-N (BON) sampling offer a simple yet effective alternative to improve performance.<n>We propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier.
• Score: 71.92083784393418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While AI agents have shown remarkable performance at various tasks, they still struggle with complex multi-modal applications, structured generation and strategic planning. Improvements via standard fine-tuning is often impractical, as solving agentic tasks usually relies on black box API access without control over model parameters. Inference-time methods such as Best-of-N (BON) sampling offer a simple yet effective alternative to improve performance. However, BON lacks iterative feedback integration mechanism. Hence, we propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier. IAD differs in how feedback is designed and integrated, specifically optimized to extract maximal signal from reward scores. We conduct a detailed comparison of baselines across key metrics on Sketch2Code, Text2SQL, and Webshop where IAD consistently outperforms baselines, achieving 3--6% absolute gains on Sketch2Code and Text2SQL (with and without LLM judges) and 8--10% gains on Webshop across multiple metrics. To better understand the source of IAD's gains, we perform controlled experiments to disentangle the effect of adaptive feedback from stochastic sampling, and find that IAD's improvements are primarily driven by verifier-guided refinement, not merely sampling diversity. We also show that both IAD and BON exhibit inference-time scaling with increased compute when guided by an optimal verifier. Our analysis highlights the critical role of verifier quality in effective inference-time optimization and examines the impact of noisy and sparse rewards on scaling behavior. Together, these findings offer key insights into the trade-offs and principles of effective inference-time optimization.

Related papers

• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [70.01883340129204]
  Single-Pass.<n>with Reference-Guided Evaluation (SPARE)<n>Novel structured framework that enables single-pass, per-step annotation by aligning each solution step to one or multiple steps in a reference solution, accompanied by explicit reasoning for evaluation.<n>SPARE achieves competitive performance on challenging mathematical datasets while offering 2.6 times greater efficiency, requiring only 38% of the runtime.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• Understanding Software Engineering Agents Through the Lens of Traceability: An Empirical Study [15.97770416681533]
  Software engineering agents (SWE agents) operate autonomously by interpreting user input and responding to environmental feedback.<n>We present the first systematic study of SWE agent behavior through the lens of execution traces.
  arXiv  Detail & Related papers  (2025-06-10T00:41:54Z)
• ProRefine: Inference-Time Prompt Refinement with Textual Feedback [8.261243439474322]
  AgenticRefine, where multiple AI agents collaborate to accomplish complex tasks like reasoning or planning, play a substantial role in many cutting-edge commercial applications.<n>We introduce ProRefine, an innovative inference-time optimization method that uses an agentic loop of LLMs to generate and apply textual feedback.<n>ProRefine significantly surpasses zero-shot Chain-of-Thought baselines by 3 to 37 percentage points.
  arXiv  Detail & Related papers  (2025-06-05T17:52:30Z)
• Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows [41.97051158610974]
  We present a systematic analysis of agentic robustness under deceptive or misleading feedback.<n>We reveal that even strongest agents are vulnerable to persuasive yet flawed critiques.<n>Our findings highlight fundamental vulnerabilities in feedback-based robustness and offer guidance for building more robust agentic systems.
  arXiv  Detail & Related papers  (2025-06-03T19:26:23Z)
• DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal [55.13854171147104]
  Large Language Models (LLMs) have revolutionized various domains, including natural language processing, data analysis, and software development.<n>We present Dynamic Action Re-Sampling (DARS), a novel inference time compute scaling approach for coding agents.<n>We evaluate our approach on SWE-Bench Lite benchmark, demonstrating that this scaling strategy achieves a pass@k score of 55% with Claude 3.5 Sonnet V2.
  arXiv  Detail & Related papers  (2025-03-18T14:02:59Z)
• Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization [9.03028904066824]
  Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes.<n>We present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation.
  arXiv  Detail & Related papers  (2025-01-23T18:59:30Z)
• Can foundation models actively gather information in interactive environments to test hypotheses? [56.651636971591536]
  We introduce a framework in which a model must determine the factors influencing a hidden reward function.<n>We investigate whether approaches such as self- throughput and increased inference time improve information gathering efficiency.
  arXiv  Detail & Related papers  (2024-12-09T12:27:21Z)
• Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision [120.40788744292739]
  We propose a two-player paradigm that separates the roles of reasoning and critique models. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time.
  arXiv  Detail & Related papers  (2024-11-25T17:11:54Z)
• Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness [27.43137305486112]
  We propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-26T12:37:26Z)
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline [78.00363373925758]
  We study the features of the BPR model, indicating their impact on its performance, and investigate open-source BPR implementations. Our analysis reveals inconsistencies between these implementations and the original BPR paper, leading to a significant decrease in performance of up to 50% for specific implementations. We show that the BPR model can achieve performance levels close to state-of-the-art methods on the top-n recommendation tasks and even outperform them on specific datasets.
  arXiv  Detail & Related papers  (2024-09-21T18:39:53Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation [67.88747330066049]
  Fine-grained feedback captures nuanced distinctions in image quality and prompt-alignment. We show that demonstrating its superiority to coarse-grained feedback is not automatic. We identify key challenges in eliciting and utilizing fine-grained feedback.
  arXiv  Detail & Related papers  (2024-06-24T17:19:34Z)
• Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning [55.96599486604344]
  We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process. We use Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data.
  arXiv  Detail & Related papers  (2024-05-01T11:10:24Z)
• Constructive Large Language Models Alignment with Diverse Feedback [76.9578950893839]
  We introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance large language models alignment. We exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data.
  arXiv  Detail & Related papers  (2023-10-10T09:20:14Z)
• Better Practices for Domain Adaptation [62.70267990659201]
  Domain adaptation (DA) aims to provide frameworks for adapting models to deployment data without using labels. Unclear validation protocol for DA has led to bad practices in the literature. We show challenges across all three branches of domain adaptation methodology.
  arXiv  Detail & Related papers  (2023-09-07T17:44:18Z)
• Confidence-Aware Active Feedback for Efficient Instance Search [21.8172170825049]
  Relevance feedback is widely used in instance search (INS) tasks to further refine imperfect ranking results. We propose a confidence-aware active feedback (CAAF) method that can efficiently select the most valuable feedback candidates. In particular, CAAF outperforms the first-place record in the public large-scale video INS evaluation of TRECVID 2021.
  arXiv  Detail & Related papers  (2021-10-23T16:14:03Z)
• MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning [36.14516028564416]
  This paper proposes an innovative Multiple Model Kalman Temporal Difference (MM-KTD) framework to learn optimal control policies. An active learning method is proposed to enhance the sampling efficiency of the system. Experimental results show superiority of the MM-KTD framework in comparison to its state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2020-05-30T06:39:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.