UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

• URL: http://arxiv.org/abs/2608.15930v1
• Date: Sun, 16 Aug 2026 20:59:59 GMT
• Title: UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations
• Abstract summary: UI-Mate is a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning.<n>It sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld and 66.2% on WindowsAgentArena.
• Score: 68.29650478987561
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.

Related papers

• EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants [10.510136510866099]
  Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve.<n>We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios.<n>We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs.
  arXiv  Detail & Related papers  (2026-08-29T17:54:02Z)
• ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions [10.090084621839088]
  We argue that screenshot-and-click is an inefficient interface for software-operating agents.<n>We introduce ASIL, an agent-native interface that exposes software through structured screenshot observations and code-executable semantic actions.<n>ASIL reaches above 80 with closed models while executing fewer than five actions per task.
  arXiv  Detail & Related papers  (2026-08-27T11:38:19Z)
• Qwen-CUA: Native Computer Use for (almost) Everything [104.78368489343713]
  We introduce Qwen-CUA, a native computer-use agent with a mixture-of-experts backbone.<n>It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs.<n>Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence.
  arXiv  Detail & Related papers  (2026-08-03T15:04:20Z)
• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents [27.09949595397669]
  Qwen-UI-Agent is a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments.<n>Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn.<n>Online RL supports training on trajectories exceeding 100 turns, with over 10,000 environments accelerating rollout.
  arXiv  Detail & Related papers  (2026-07-30T13:58:41Z)
• ProCUA-SFT Technical Report [80.97543110323542]
  Training computer-use agents (CUAs) requires large-scale, diverse trajectory data collected in full desktop environments.<n>We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories.<n>Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model.
  arXiv  Detail & Related papers  (2026-06-15T22:04:11Z)
• Terminal-World: Scaling Terminal-Agent Environments via Agent Skills [52.39713754337834]
  We introduce Terminal-World, a fully automated pipeline that uses agent skills as the central synthesis primitive.<n>We construct 5,723 training environments and train Terminal-World-8B/14B/32B, evaluated across 6 benchmarks.<n>Terminal-World-32B surpasses Nemotron-Terminal-32B on Terminal-Bench 2.0 by +4.5 Pass@1 (31.5) and achieves 43.8 Pass@3.
  arXiv  Detail & Related papers  (2026-05-20T08:14:51Z)
• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents [56.72789202127874]
  The paper introduces GUI-Owl-1.5, the latest native GUI agent model.<n>It supports a range of platforms (desktop, mobile, browser, and more) to enable cloud-edge collaboration and real-time interaction.<n>It achieves state-of-the-art results on more than 20+ GUI benchmarks on open-source models.
  arXiv  Detail & Related papers  (2026-02-15T01:52:19Z)
• MAI-UI Technical Report: Real-World Centric Foundation GUI Agents [33.46555542782679]
  MAI-UI is a family of foundation GUI agents spanning the full spectrum of sizes, including 2B, 8B, 32B, and 235B-A22B variants.<n>We identify four key challenges to realistic deployment: the lack of native agent-user interaction, the limits of UI-only operation, and the absence of a practical deployment architecture.
  arXiv  Detail & Related papers  (2025-12-26T14:51:52Z)
• UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning [51.54456545661045]
  We introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives.<n>To achieve this, we propose a two-stage training framework: supervised fine-tuning and reinforcement learning.<n>Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks.
  arXiv  Detail & Related papers  (2025-10-23T07:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.