OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

• URL: http://arxiv.org/abs/2607.28609v2
• Date: Thu, 06 Aug 2026 17:55:59 GMT
• Title: OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
• Abstract summary: Computer-using agents (CUAs) are advancing rapidly across the digital world.<n>The field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories.<n>We introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories.
• Score: 87.08963239083614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60x lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.

Related papers

• SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents [43.62427340281939]
  Rule-based evaluation struggles to align with human intention and goes stale when an app updates or its online content drifts.<n>We propose the textbfSeekJudge framework, in which four role-specialized agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict.<n>SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL.
  arXiv  Detail & Related papers  (2026-07-25T16:00:45Z)
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents [51.523913302114266]
  TRACE (Turn-level Reward Assignment via Credit Estimation) is a dense credit-assignment method for agentic reinforcement learning.<n>It represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values.<n>It substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data.
  arXiv  Detail & Related papers  (2026-07-15T16:16:42Z)
• 3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects [17.49934543489464]
  We introduce 3D-DefectBench, a benchmark and framework for analysis of 3D defect detection pipelines.<n>Using a balanced factorial design, we vary four pipeline factors, VLM, camera protocol, visual input, and prompt schema, across 84 inference designs.<n>The best of 12 VLM judges still lag behind trained human labelers, while texture agreement drops sharply when expert-consensus labels are replaced by noisier silver labels.
  arXiv  Detail & Related papers  (2026-07-12T16:41:24Z)
• Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation [15.119045051735633]
  We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents.<n>Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator.<n>Experiments show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning.
  arXiv  Detail & Related papers  (2026-06-23T12:46:29Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering [0.0]
  Large Language Models are increasingly used as judges to evaluate code artifacts.<n>Current practice lacks a principled account of reliability and bias.<n>This paper studies LLM-as-a-Judge for code through a measurement-first lens.
  arXiv  Detail & Related papers  (2026-04-18T02:35:05Z)
• OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards [44.03496012544118]
  OS-Themis is a scalable and accurate multi-agent critic framework.<n>It decomposes trajectories into verifiable milestones to isolate critical evidence for decision making.<n>It employs a review mechanism to strictly audit the evidence chain before making the final verdict.<n>Experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training.
  arXiv  Detail & Related papers  (2026-03-19T17:47:47Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• One Token to Fool LLM-as-a-Judge [52.45386385722788]
  Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models.<n>We uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking.
  arXiv  Detail & Related papers  (2025-07-11T17:55:22Z)
• Revisiting Few-Shot Object Detection with Vision-Language Models [49.79495118650838]
  We revisit the task of few-shot object detection (FSOD) in the context of recent foundational vision-language models (VLMs) We propose Foundational FSOD, a new benchmark protocol that evaluates detectors pre-trained on any external data. We discuss our recent CVPR 2024 Foundational FSOD competition and share insights from the community.
  arXiv  Detail & Related papers  (2023-12-22T07:42:00Z)
• RobustBench: a standardized adversarial robustness benchmark [84.50044645539305]
  Key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks. We analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
  arXiv  Detail & Related papers  (2020-10-19T17:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.