ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

• URL: http://arxiv.org/abs/2608.26991v1
• Date: Thu, 27 Aug 2026 11:38:19 GMT
• Title: ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
• Abstract summary: We argue that screenshot-and-click is an inefficient interface for software-operating agents.<n>We introduce ASIL, an agent-native interface that exposes software through structured screenshot observations and code-executable semantic actions.<n>ASIL reaches above 80 with closed models while executing fewer than five actions per task.
• Score: 10.090084621839088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

Related papers

• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations [68.29650478987561]
  UI-Mate is a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning.<n>It sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld and 66.2% on WindowsAgentArena.
  arXiv  Detail & Related papers  (2026-08-16T20:59:59Z)
• Qwen-CUA: Native Computer Use for (almost) Everything [104.78368489343713]
  We introduce Qwen-CUA, a native computer-use agent with a mixture-of-experts backbone.<n>It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs.<n>Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence.
  arXiv  Detail & Related papers  (2026-08-03T15:04:20Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? [29.5539656241093]
  Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments.<n>Existing benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions.<n>We introduce a benchmark built on 23 deployable systems across six professional domains, containing 106 tasks grounded in realistic work scenarios.<n>Experiments show that representative LLM-based agents struggle on-Bench, with even the strongest model completing fewer than 4% of tasks end-to-end.
  arXiv  Detail & Related papers  (2026-05-15T09:35:15Z)
• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents [56.72789202127874]
  The paper introduces GUI-Owl-1.5, the latest native GUI agent model.<n>It supports a range of platforms (desktop, mobile, browser, and more) to enable cloud-edge collaboration and real-time interaction.<n>It achieves state-of-the-art results on more than 20+ GUI benchmarks on open-source models.
  arXiv  Detail & Related papers  (2026-02-15T01:52:19Z)
• A Case for Declarative LLM-friendly Interfaces for Improved Efficiency of Computer-Use Agents [7.687846129934911]
  We propose Goal-Oriented Interface (GOI), a novel abstraction that transforms existing GUIs into three declarative primitives.<n>GOI does not require modifying the application source code or relying on application programming interfaces (APIs)
  arXiv  Detail & Related papers  (2025-10-06T09:14:58Z)
• Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces [5.258138614911196]
  We introduce a lightweight hierarchical reinforcement learning framework, ComputerAgent, that formulates OS control as a two-level option process.<n>On a suite of 135 real-world desktop tasks, ComputerAgent attains 92.1% success on simple tasks and 58.8% on hard tasks.<n>Results demonstrate that hierarchical RL offers a practical, scalable alternative to monolithic MLLM-based automation for computer control.
  arXiv  Detail & Related papers  (2025-09-22T13:14:47Z)
• Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
  This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
  arXiv  Detail & Related papers  (2025-08-21T00:39:12Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)
• You Only Look at Screens: Multimodal Chain-of-Action Agents [37.118034745972956]
  Auto-GUI is a multimodal solution that directly interacts with the interface. We propose a chain-of-action technique to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions.
  arXiv  Detail & Related papers  (2023-09-20T16:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.