Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

• URL: http://arxiv.org/abs/2606.24515v1
• Date: Tue, 23 Jun 2026 12:46:29 GMT
• Title: Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation
• Abstract summary: We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents.<n>Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator.<n>Experiments show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning.
• Score: 15.119045051735633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because open-ended desktop environments rarely provide scalable, machine-readable reward signals: task success is often visually grounded and hard to specify with handcrafted reward functions or dense manual labels. We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents. Given a final screenshot and the original instruction, a Vision-Language Model judges task completion and provides terminal feedback without task-specific heuristics or manual labels during policy optimization. Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning. These results suggest that autonomous evaluation can serve as a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.

Related papers

• GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis [42.18402261450695]
  GSAR (Goal-State-Anchor Reward) is a RL reward framework that supports scalable task generation and delivers reliable reward signals.<n>Our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods.
  arXiv  Detail & Related papers  (2026-08-24T06:27:47Z)
• OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models [87.08963239083614]
  Computer-using agents (CUAs) are advancing rapidly across the digital world.<n>The field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories.<n>We introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories.
  arXiv  Detail & Related papers  (2026-07-30T17:57:41Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• RewardHarness: Self-Evolving Agentic Post-Training [51.16917192137169]
  RewardHarness is a self-evolving agentic reward framework for instruction-guided image editing.<n>It aligns with human preferences by iteratively evolving a library of tools and skills from as few as 100 preference demonstrations.<n>It achieves 47.4% average accuracy on image-editing evaluation benchmarks, surpassing GPT-5 by 5.3 points.
  arXiv  Detail & Related papers  (2026-05-09T05:32:48Z)
• OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards [44.03496012544118]
  OS-Themis is a scalable and accurate multi-agent critic framework.<n>It decomposes trajectories into verifiable milestones to isolate critical evidence for decision making.<n>It employs a review mechanism to strictly audit the evidence chain before making the final verdict.<n>Experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training.
  arXiv  Detail & Related papers  (2026-03-19T17:47:47Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction [7.731207237810125]
  VAGEN is a framework that employs a verifier agent equipped with interaction tools to autonomously plan verification strategies.<n>We show that VAGEN significantly improves evaluation accuracy compared to LLM-as-a-Judge baselines.
  arXiv  Detail & Related papers  (2026-01-31T07:36:54Z)
• SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models [42.89413870143421]
  Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations.<n>Current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity.<n>We propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework.
  arXiv  Detail & Related papers  (2025-11-19T16:52:23Z)
• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration [24.005812534024965]
  ProRe is a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents.<n>ProRe improves reward accuracy and F1 score by up to 5.3% and 19.4%, respectively.
  arXiv  Detail & Related papers  (2025-09-26T03:29:36Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• Learning to Reason without External Rewards [100.27210579418562]
  Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision.<n>We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data.<n>We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal.
  arXiv  Detail & Related papers  (2025-05-26T07:01:06Z)
• VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences [13.337649128532307]
  Preference-based RL mitigates some of these pitfalls by learning rewards from comparative feedback.<n>A single final-state image generally fails to capture the agent's full motion.<n>We present a two-part solution that both improves feedback accuracy and better aligns reward learning with the agent's policy.
  arXiv  Detail & Related papers  (2025-03-18T01:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.