Related papers: Intrinsic Credit Assignment for Long Horizon Interaction

Related papers

Agentic Reinforcement Learning with Self-Distilled Reward Shaping [12.091871903119698]
Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit.<n>We introduce Agentic Reinforcement Learning with Self-Distilled Reward Shaping, a framework for constructing return-associated token-level credit for language agents.<n>Experiments show that ADRS consistently improves performance on long-horizon tasks, with gains persisting across RL backbones, reduced-data settings, unseen tasks, and extended training.
arXiv Detail & Related papers (2026-08-04T06:56:47Z)
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents [24.215311212134882]
We propose Reward-Swap Policy Optimization (RSPO) to leverage the rich information from dense process rewards to facilitate training with outcome rewards.<n>RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards.
arXiv Detail & Related papers (2026-07-06T06:32:39Z)
Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
arXiv Detail & Related papers (2026-06-05T15:09:52Z)
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation [40.17737666526493]
We propose PACT, a Preference-calibrated Actor-Critic Training framework.<n>We first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction.<n>We then build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment.
arXiv Detail & Related papers (2026-06-02T17:38:25Z)
Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
arXiv Detail & Related papers (2026-04-13T15:18:51Z)
SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents [0.31034395048547575]
Existing approaches rely on static prompting strategies or handcrafteds to select actions such as code editing, file navigation, and test execution.<n>We propose SWE-Shepherd, a framework that introduces Process Reward Models (PRMs) to provide dense, step-level supervision for repository-level code agents.<n>Experiments on SWE-Bench Verified demonstrate improved interaction efficiency and action quality, while also highlighting challenges in aligning intermediate rewards with final task success.
arXiv Detail & Related papers (2026-04-12T06:51:47Z)
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning [47.756127177035246]
We propose (HISR) exploiting Hindsight Information to modulate Segmental process Rewards.<n>Specifically, a segment-level process RM is presented to assign rewards for each sub-goal in the task.<n>To emphasize significant segments in the trajectory, a hindsight model is devised to reflect the preference of performing a certain action after knowing the trajectory outcome.
arXiv Detail & Related papers (2026-03-19T09:42:05Z)
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction [14.099864893966377]
Reinforcement learning post-training reduces output diversity by exploiting coherence-based rewards.<n>This collapse is especially harmful in live jamming, where musical creativity relies on dynamic variation and mutual responsiveness.<n>We propose a novel adversarial training method on policy-generated trajectories to mitigate reward hacking in RL post-training for melody-to-chord accompaniment.
arXiv Detail & Related papers (2025-11-22T02:12:41Z)
Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
arXiv Detail & Related papers (2025-09-23T16:15:42Z)
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning [87.7836502955847]
We propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning.<n>Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood.<n>We introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy.
arXiv Detail & Related papers (2025-06-10T12:40:39Z)
Augmenting Unsupervised Reinforcement Learning with Self-Reference [63.68018737038331]
Humans possess the ability to draw on past experiences explicitly when learning new tasks. We propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information. Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark.
arXiv Detail & Related papers (2023-11-16T09:07:34Z)
Credit-cognisant reinforcement learning for multi-agent cooperation [0.0]
We introduce the concept of credit-cognisant rewards, which allows an agent to perceive the effect its actions had on the environment as well as on its co-agents. We show that by manipulating these experiences and constructing the reward contained within them to include the rewards received by all the agents within the same action sequence, we are able to improve significantly on the performance of independent deep Q-learning.
arXiv Detail & Related papers (2022-11-18T09:00:25Z)
Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
arXiv Detail & Related papers (2022-08-09T17:29:49Z)
Targeted Data Acquisition for Evolving Negotiation Agents [6.953246373478702]
Successful negotiators must learn how to balance optimizing for self-interest and cooperation. Current artificial negotiation agents often heavily depend on the quality of the static datasets they were trained on. We introduce a targeted data acquisition framework where we guide the exploration of a reinforcement learning agent.
arXiv Detail & Related papers (2021-06-14T19:45:59Z)
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
arXiv Detail & Related papers (2021-06-09T14:10:50Z)
Bridging the Imitation Gap by Adaptive Insubordination [88.35564081175642]
We show that when the teaching agent makes decisions with access to privileged information, this information is marginalized during imitation learning. We propose 'Adaptive Insubordination' (ADVISOR) to address this gap. ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration.
arXiv Detail & Related papers (2020-07-23T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.