Related papers: ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

URL: http://arxiv.org/abs/2608.13622v1
Date: Thu, 13 Aug 2026 01:51:53 GMT
Title: ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction
Abstract summary: ARC is a novel paradigm for responsive, steerable, and execution-aware user-agent interaction.<n> ARC substantially strengthens the core $/2$ tool-use benchmarks.<n>inter reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline.
Score: 4.6344273009233214
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $τ/τ^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.

Related papers

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL [6.238212335422264]
Group-relative reinforcement learning waits for sibling rollouts of the same prompt.<n>We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor.<n>We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order.
arXiv Detail & Related papers (2026-08-25T17:52:19Z)
Rank-Then-Act: Reward-Free Control from Frame-Order Progress [5.357451930622806]
Rank-Then-Act (RTA) is a framework for learning control policies from expert video demonstrations without environment rewards.<n>We train a Vision-Language Model (VLM) offline as a progress-based ordinal scorer.<n>We compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal.
arXiv Detail & Related papers (2026-07-02T08:50:32Z)
TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
arXiv Detail & Related papers (2026-06-09T17:16:03Z)
A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
arXiv Detail & Related papers (2026-03-04T07:23:54Z)
Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers [55.33468902405567]
We propose a new learning paradigm, In-Context Preference-based Reinforcement Learning (ICPRL), in which both pretraining and deployment rely solely on preference feedback.<n>ICPRL enables strong in-context generalization to unseen tasks, achieving performance comparable to ICRL methods trained with full reward supervision.
arXiv Detail & Related papers (2026-02-09T03:42:16Z)
CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models [29.703793991791674]
Existing RAG-oriented reinforcement learning methods rely on external rewards that often fail to evaluate document faithfulness.<n>We propose a novel "internal-external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR)<n>CLR directly optimize the log-likelihood gap between responses conditioned on prompts with and without supporting evidence.
arXiv Detail & Related papers (2026-02-02T12:21:59Z)
Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval [49.85856484781787]
We introduce Interact-RAG, a new paradigm that elevates the LLM agent into an active manipulator of the retrieval process.<n>We develop a reasoning-enhanced workflow, which enables both zero-shot execution and the synthesis of interaction trajectories.<n>Experiments across six benchmarks demonstrate that Interact-RAG significantly outperforms other advanced methods.
arXiv Detail & Related papers (2025-10-31T15:48:43Z)
Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
arXiv Detail & Related papers (2025-09-23T16:15:42Z)
STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
arXiv Detail & Related papers (2025-08-26T08:47:58Z)
Resisting Contextual Interference in RAG via Parametric-Knowledge Reinforcement [9.66890519317288]
Retrieval-augmented generation (RAG) improves performance on knowledge-intensive tasks but can be derailed by wrong, irrelevant, or conflicting retrieved text.<n>We propose Knowledgeable-R1, a reinforcement-learning framework that explicitly trains large language models to use parametric knowledge.
arXiv Detail & Related papers (2025-06-05T15:34:15Z)
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
arXiv Detail & Related papers (2024-08-14T11:29:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.