Asymmetric Actor-Critic for Multi-turn LLM Agents

• URL: http://arxiv.org/abs/2604.00304v1
• Date: Tue, 31 Mar 2026 22:56:21 GMT
• Title: Asymmetric Actor-Critic for Multi-turn LLM Agents
• Abstract summary: We propose an asymmetric actor-critic framework for reliable conversational agents.<n>A powerful proprietary LLM acts as the actor, while a smaller open-source critic provides runtime supervision.<n>We show that our approach significantly improves reliability and task success over strong single-agent baselines.
• Score: 50.245019205783855
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) exhibit strong reasoning and conversational abilities, but ensuring reliable behavior in multi-turn interactions remains challenging. In many real-world applications, agents must succeed in one-shot settings where retries are impossible. Existing approaches either rely on reflection or post-hoc evaluation, which require additional attempts, or assume fully trainable models that cannot leverage proprietary LLMs. We propose an asymmetric actor-critic framework for reliable conversational agents. A powerful proprietary LLM acts as the actor, while a smaller open-source critic provides runtime supervision, monitoring the actor's actions and intervening within the same interaction trajectory. Unlike training-based actor-critic methods, our framework supervises a fixed actor operating in open-ended conversational environments. The design leverages a generation-verification asymmetry: while high-quality generation requires large models, effective oversight can often be achieved by smaller ones. We further introduce a data generation pipeline that produces supervision signals for critic fine-tuning without modifying the actor. Experiments on $τ$-bench and UserBench show that our approach significantly improves reliability and task success over strong single-agent baselines. Moreover, lightweight open-source critics rival or surpass larger proprietary models in the critic role, and critic fine-tuning yields additional gains over several state-of-the-art methods.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents [12.20001133898725]
  LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making.<n>Existing approaches often rely on an expert solver or a critic that generates task-specific corrections.<n>We introduce Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention.
  arXiv  Detail & Related papers  (2026-08-21T12:20:14Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings [2.9017174137438815]
  Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function.<n>Large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities.<n>We propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model.
  arXiv  Detail & Related papers  (2026-03-16T15:29:41Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• Actor-Critic without Actor [4.94481688445056]
  We introduce Actor-Critic without Actor (ACA), a lightweight framework that eliminates the explicit actor network and instead generates actions directly from the field of a noise-level critic.<n>ACA achieves more favorable learning curves and competitive performance compared to both standard actor-critic and state-of-the-art diffusion-based methods.
  arXiv  Detail & Related papers  (2025-09-25T11:33:09Z)
• The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement [61.00950725408354]
  Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions.<n>In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback.
  arXiv  Detail & Related papers  (2025-03-20T10:42:33Z)
• Sparse Rewards Can Self-Train Dialogue Agents [22.799506097310008]
  We introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback.<n>We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ.<n>We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-09-06T21:00:57Z)
• DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics [52.242449026151846]
  Multi-agent debates have been introduced to improve the accuracy of Large Language Models (LLMs)<n>We propose DebUnc, a debate framework that uses uncertainty metrics to assess agent confidence.
  arXiv  Detail & Related papers  (2024-07-08T22:15:01Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)
• Solving Continuous Control via Q-learning [54.05120662838286]
  We show that a simple modification of deep Q-learning largely alleviates issues with actor-critic methods. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods.
  arXiv  Detail & Related papers  (2022-10-22T22:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.