OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

• URL: http://arxiv.org/abs/2603.19191v1
• Date: Thu, 19 Mar 2026 17:47:47 GMT
• Title: OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards
• Abstract summary: OS-Themis is a scalable and accurate multi-agent critic framework.<n>It decomposes trajectories into verifiable milestones to isolate critical evidence for decision making.<n>It employs a review mechanism to strictly audit the evidence chain before making the final verdict.<n>Experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training.
• Score: 44.03496012544118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has the potential to improve the robustness of GUI agents in stochastic environments, yet training is highly sensitive to the quality of the reward function. Existing reward approaches struggle to achieve both scalability and performance. To address this, we propose OS-Themis, a scalable and accurate multi-agent critic framework. Unlike a single judge, OS-Themis decomposes trajectories into verifiable milestones to isolate critical evidence for decision making and employs a review mechanism to strictly audit the evidence chain before making the final verdict. To facilitate evaluation, we further introduce OmniGUIRewardBench (OGRBench), a holistic cross-platform benchmark for GUI outcome rewards, where all evaluated models achieve their best performance under OS-Themis. Extensive experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training, and a 6.9% gain when used for trajectory validation and filtering in the self-training loop, highlighting its potential to drive agent evolution.

Related papers

• GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis [42.18402261450695]
  GSAR (Goal-State-Anchor Reward) is a RL reward framework that supports scalable task generation and delivers reliable reward signals.<n>Our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods.
  arXiv  Detail & Related papers  (2026-08-24T06:27:47Z)
• Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation [15.119045051735633]
  We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents.<n>Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator.<n>Experiments show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning.
  arXiv  Detail & Related papers  (2026-06-23T12:46:29Z)
• GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models [18.14416135619429]
  We propose the GUI Action Critic's Data Flywheel System (GAIA), a training framework that enables the models to have iterative critic capabilities.<n>Specifically, we train an Intuitive Critic Model (ICM) using positive and negative action examples from a base agent.<n>We conduct experiments on various datasets and demonstrate that the proposed ICM can improve the test-time performance of various closed-source and open-source models.
  arXiv  Detail & Related papers  (2026-01-26T06:29:41Z)
• GDRO: Group-level Reward Post-training Suitable for Diffusion Models [55.948229011478304]
  Group-level rewards successfully align the model with the targeted reward.<n>Group-level Direct Reward Optimization (GDRO) is a new post-training paradigm for group-level reward alignment.<n>GDRO supports full offline training that saves the large time cost for image rollout sampling.<n>It is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtainity.
  arXiv  Detail & Related papers  (2026-01-05T11:47:18Z)
• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models [54.44308299945632]
  We introduce OS-Oracle that makes three core contributions: a scalable data pipeline for cross-platform GUI critic data; a two-stage training paradigm combining supervised fine-tuning and consistency-preserving group relative policy optimization; and OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms.<n>The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain.
  arXiv  Detail & Related papers  (2025-12-18T08:29:50Z)
• A Mathematical Framework for Custom Reward Functions in Job Application Evaluation using Reinforcement Learning [0.8709142317087694]
  This article describes a new two-step process to design a more refined resume evaluation model based on a small language model.<n>The model is finetuned using GRPO on a custom reward function.<n>Our resulting GRPO-polished model demonstrates significant real-world efficacy, achieving a final accuracy of 91% on unseen test data.
  arXiv  Detail & Related papers  (2025-11-20T06:06:30Z)
• GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks [75.50160982584943]
  We introduce a Process Reward Model that provides dense, step-by-step feedback to guide agents.<n>Gui-Shepherd is trained on a diverse large-scale data set of $52$k interactions.<n>We are the first to conduct a systematic study of process supervision in GUI agents.
  arXiv  Detail & Related papers  (2025-09-28T08:35:16Z)
• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration [24.005812534024965]
  ProRe is a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents.<n>ProRe improves reward accuracy and F1 score by up to 5.3% and 19.4%, respectively.
  arXiv  Detail & Related papers  (2025-09-26T03:29:36Z)
• CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks [11.121687042616974]
  Reinforcement Learning (RL) can effectively enhance agents' performance in dynamic interactive GUI environments.<n>Most approaches collapse task-specific nuances into a single, coarse reward, leaving the agent with a uniform signal that yields inefficient policy updates.<n>We propose CRAFT-GUI, a curriculum learning framework based on Group Relative Policy Optimization (GRPO) that explicitly accounts for the varying difficulty across trajectories.
  arXiv  Detail & Related papers  (2025-08-15T09:55:02Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.