ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents

• URL: http://arxiv.org/abs/2606.22948v1
• Date: Mon, 22 Jun 2026 07:28:55 GMT
• Title: ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
• Abstract summary: Environment-Native Verified Search builds verified supervision before policy optimization.<n> OSWorld-Noisy is a benchmark for recoverable desktop interruptions.<n> ENVS reaches 30.3 pass@8 on original evaluations and 29.0 on OSWorld-Noisy, outperforming ARPO-style online searches.
• Score: 35.35193712019351
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As multimodal agents move from interface understanding to real software control, successful trajectory discovery in live desktop environments becomes a key challenge. GUI tasks require long-horizon sequences of precise mouse and keyboard actions, while feedback is sparse, delayed, and costly to obtain through VM rollouts. We propose Environment-Native Verified Search (ENVS), a training-time search-and-filter pipeline that uses the environment to construct verified supervision before policy optimization: it branches over behaviorally distinct GUI actions in live OSWorld VMs, verifies successful leaves, and trains from globally balanced step-level supervision. To evaluate robustness under realistic desktop interruptions, we also introduce OSWorld-Noisy, a dynamic benchmark for recoverable desktop interruptions that preserves the original tasks while testing whether agents can refocus, dismiss, wait, or recover under live perturbations. On the 300-task OSWorld pool, ENVS reaches 30.3 pass@8 on original evaluations and 29.0 on OSWorld-Noisy, outperforming matched ARPO-style online RL while reducing compute from 184-192 to 138-153 GPU-hours; even with only 30% of its search data, ENVS reaches 27.0 pass@8, exceeding ARPO from the base model. Training from noisy environments also better preserves visual-reasoning abilities on auxiliary benchmarks, including OSWorld-G Refusal (16.7 vs. 1.9) and BLINK Functional Correspondence (26.2 vs. 23.1).

Related papers

• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations [68.29650478987561]
  UI-Mate is a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning.<n>It sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld and 66.2% on WindowsAgentArena.
  arXiv  Detail & Related papers  (2026-08-16T20:59:59Z)
• Qwen-CUA: Native Computer Use for (almost) Everything [104.78368489343713]
  We introduce Qwen-CUA, a native computer-use agent with a mixture-of-experts backbone.<n>It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs.<n>Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence.
  arXiv  Detail & Related papers  (2026-08-03T15:04:20Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents [74.07551833809536]
  We introduce LiteCoder-Terminal-Gen, a zero-dependency pipeline that generates terminal training environments directly from domain specifications.<n>Using this framework, we construct two large-scale resources: LiteCoder-Terminal-SFT, comprising 11,255 expert trajectories across 10 domains, and LiteCoder-Terminal-RL, featuring 602 verifiable environments for trajectory-level preference optimization.
  arXiv  Detail & Related papers  (2026-05-28T08:11:57Z)
• GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training [30.675339113121225]
  We present GUI-GENESIS, a framework to automatically synthesize efficient GUI training environments with verifiable rewards.<n>Experiments show that GUI-GENESIS reduces environment latency by 10 times and costs by over $28,000 per epoch compared to training on real applications.
  arXiv  Detail & Related papers  (2026-02-15T10:58:01Z)
• ANCHOR: Branch-Point Data Generation for GUI Agents [52.22377425487]
  End-to-end GUI agents for real desktop environments require large amounts of high-quality interaction data.<n>We present a trajectory expansion framework Anchor that bootstraps scalable desktop supervision from a small set of verified seed demonstrations.<n>Experiments on standard desktop benchmarks, OSWorld and WindowsAgentArena, show that models fine-tuned on our expanded corpus achieve consistent improvements.
  arXiv  Detail & Related papers  (2026-02-06T19:55:26Z)
• AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents [36.66219528445988]
  We introduce AndroidLens, a challenging evaluation framework for mobile GUI agents.<n>It comprises 571 long-latency tasks in both Chinese and English environments.<n>Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP.
  arXiv  Detail & Related papers  (2025-12-24T17:40:42Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
  The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
  arXiv  Detail & Related papers  (2025-09-02T17:44:45Z)
• WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point [17.165899818213475]
  We introduce WorldGUI, a comprehensive GUI benchmark containing tasks across ten widely used desktop and web applications.<n>WorldGUI-Agent is a universal framework that unifies three core modules: Planner-Critic for high-level plan refinement, Step-Check for intermediate verification, and Actor-Critic for action-level optimization.
  arXiv  Detail & Related papers  (2025-02-12T01:06:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.