OpenClaw-RL: Train Any Agent Simply by Talking

• URL: http://arxiv.org/abs/2603.10165v1
• Date: Tue, 10 Mar 2026 18:59:01 GMT
• Title: OpenClaw-RL: Train Any Agent Simply by Talking
• Abstract summary: Next-state signals are universal, and policy can learn from all of them simultaneously.<n>Personal conversations, terminal executions, GUI interactions, SWE tasks, and tool-call traces are not separate training problems.<n>OpenClaw-RL enables an agent to improve simply by being used, recovering conversational signals from user re-queries, corrections, and explicit feedback.
• Score: 54.06773485601523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Every agent interaction generates a next-state signal, namely the user reply, tool output, terminal or GUI state change that follows each action, yet no existing agentic RL system recovers it as a live, online learning source. We present OpenClaw-RL, a framework built on a simple observation: next-state signals are universal, and policy can learn from all of them simultaneously. Personal conversations, terminal executions, GUI interactions, SWE tasks, and tool-call traces are not separate training problems. They are all interactions that can be used to train the same policy in the same loop. Next-state signals encode two forms of information: evaluative signals, which indicate how well the action performed and are extracted as scalar rewards via a PRM judge; and directive signals, which indicate how the action should have been different and are recovered through Hindsight-Guided On-Policy Distillation (OPD). We extract textual hints from the next state, construct an enhanced teacher context, and provide token-level directional advantage supervision that is richer than any scalar reward. Due to the asynchronous design, the model serves live requests, the PRM judges ongoing interactions, and the trainer updates the policy at the same time, with zero coordination overhead between them. Applied to personal agents, OpenClaw-RL enables an agent to improve simply by being used, recovering conversational signals from user re-queries, corrections, and explicit feedback. Applied to general agents, the same infrastructure supports scalable RL across terminal, GUI, SWE, and tool-call settings, where we additionally demonstrate the utility of process rewards. Code: https://github.com/Gen-Verse/OpenClaw-RL

Related papers

• OpenForgeRL: Train Harness-native Agents in Any Environment [60.23309689179493]
  We present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments.<n>By decoupling inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with.
  arXiv  Detail & Related papers  (2026-07-23T17:38:30Z)
• SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL [38.05711358650287]
  Computer use agents (CUAs) are emerging as a powerful interface for automating complex digital through visual perception and GUI execution.<n>Online reinforcement learning with verifiable rewards (RLVR) has emerged as a key direction for scaling their capabilities.<n>We introduce ScaleCUA, a unified framework that scales online RL for CUAs via verifiable task synthesis and efficient training.
  arXiv  Detail & Related papers  (2026-07-13T07:32:35Z)
• Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning [9.576383475538606]
  We present Claw-R1, an interactive step-level data system for agentic RL.<n>Claw-R1 connects heterogeneous agent runtimes with RL training backends through two core components.<n>In our demo, users can interactively inspect live trajectories, examine the state, action, and reward of each step, curate data by quality and readiness, and configure training-ready batches.
  arXiv  Detail & Related papers  (2026-06-08T07:35:18Z)
• Will the Agent Recuse Itself? Measuring LLM-Agent Compliance with In-Band Access-Deny Signals [0.0]
  We propose a lightweight, published in-band deny signal -- the Recuse Signal.<n>It emits over a protocol's existing channels asking a connecting automated agent to withdraw voluntarily.<n>We run a controlled experiment in which fresh agents are given a benign operations task and observed for recusal.
  arXiv  Detail & Related papers  (2026-06-04T17:50:54Z)
• GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering [64.23115520219609]
  We propose a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL.<n>GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.
  arXiv  Detail & Related papers  (2026-05-28T08:28:10Z)
• RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System [52.3348044324205]
  We propose RLAnything, a reinforcement learning framework that forges environment, policy, and reward models through closed-loop optimization.<n>Specifically, the policy is trained with integrated feedback from step-wise and outcome signals.<n>Our theory-motivated automatic environment adaptation improves training for both the reward and policy models.
  arXiv  Detail & Related papers  (2026-02-02T18:59:04Z)
• From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents [23.583947864141162]
  EigenData is a hierarchical multi-agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers.<n>Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training.<n>Our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.
  arXiv  Detail & Related papers  (2026-01-30T06:01:23Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• Agent Lightning: Train ANY AI Agents with Reinforcement Learning [24.13422767414729]
  We present Agent Lightning, a framework that enables Reinforcement Learning (RL)-based training of Large Language Models (LLMs) for any AI agent.<n>By formulating agent execution as Markov decision process, we define an unified data interface and propose a hierarchical RL algorithm, LightningRL, which contains a credit assignment module.<n>For the system design, we introduce a Training-Agent Disaggregation architecture, and brings agent observability frameworks into agent runtime.
  arXiv  Detail & Related papers  (2025-08-05T17:50:13Z)
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving [26.413753656936688]
  Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation.<n>While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial.
  arXiv  Detail & Related papers  (2025-05-12T17:23:34Z)
• RLZero: Direct Policy Inference from Language Without In-Domain Supervision [40.046873614139464]
  Natural language offers an intuitive alternative for instructing reinforcement learning agents.<n>We present a new approach that uses a pretrained RL agent trained using unlabeled, offline interactions.<n>We show that components of RL can be used to generate policies zeroshot from cross-embodied videos.
  arXiv  Detail & Related papers  (2024-12-07T18:31:16Z)
• Graph Decision Transformer [83.76329715043205]
  Graph Decision Transformer (GDT) is a novel offline reinforcement learning approach. GDT models the input sequence into a causal graph to capture potential dependencies between fundamentally different concepts. Our experiments show that GDT matches or surpasses the performance of state-of-the-art offline RL methods on image-based Atari and OpenAI Gym.
  arXiv  Detail & Related papers  (2023-03-07T09:10:34Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.