Tactile: Giving Computer-Using Agents Hands and Feet

• URL: http://arxiv.org/abs/2607.14443v1
• Date: Thu, 16 Jul 2026 00:35:25 GMT
• Title: Tactile: Giving Computer-Using Agents Hands and Feet
• Abstract summary: We present Tactile, an open-source tool layer that gives agents a more reliable "hands and feet" for desktop use.<n>Tactile converts heterogeneous UI evidence-adapted-operating-system accessibility semantics, OCR-grounded text, and visual fallback regions--into action-grounded interface states.<n>Agents operate through an observe-ground-act-verify loop that prefers native semantic actions when available, falls back to OCR-grounded coordinates when visible text is the best evidence, and keeps full provenance for replay and failure attribution.
• Score: 9.655595397922488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents are becoming capable software operators, but their interface to desktop applications is still often a brittle motor layer: they look at screenshots, predict coordinates, click, and hope that the visible state changed as intended. This collapses target grounding, action execution, and outcome verification into a single ambiguous operation. We present Tactile, an open-source tool layer that gives agents a more reliable "hands and feet" for desktop use. Tactile converts heterogeneous UI evidence--operating-system accessibility semantics, OCR-grounded text, and visual fallback regions--into action-grounded interface states: compact target candidates with source labels, roles or text, state, geometry, executable affordances, and verification cues. Agents operate through an observe-ground-act-verify loop that prefers native semantic actions when available, falls back to OCR-grounded coordinates when visible text is the best evidence, and keeps full provenance for replay and failure attribution. On macOSWorld-style tasks, adding Tactile improves Codex Success@100 from 41.1% to 50.0% overall and from 45.2% to 55.3% on accessibility-adapted tasks; a 96-task cross-agent subset shows consistent gains across Codex, Claude Code, OpenCode, and Goose. These results suggest that reliable computer use requires not only stronger models, but also a reusable execution substrate that exposes software actions as semantic, verifiable, and auditable objects rather than anonymous screen coordinates.

Related papers

• EchoPath: Execution-Level Replayable Memory for GUI Agents [45.101663136838916]
  We introduce EchoPath, a model-agnostic harness that converts artifact-validated GUI trajectories into standardized, parameter-controlled callable memories.<n>Each memory stores task-intent keys, application and state preconditions, flexible input parameters, GUI evidence, validation provenance, and lifecycle state.<n>In experiments with real computer-use tasks, EchoPath reduced median token cost by more than 90% and median execution time by about 60%.
  arXiv  Detail & Related papers  (2026-09-15T04:56:17Z)
• StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents [55.650871465979066]
  StateAct is a code-first, multi-agent harness built around this distinction.<n>Its main agent works directly with program state by using code.<n>A dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it.
  arXiv  Detail & Related papers  (2026-07-24T14:17:03Z)
• PalmClaw: A Native On-Device Agent Framework for Mobile Phones [16.173016265644673]
  We present textbfPalmClaw, an open-source agent framework that runs on mobile phones and manages the sessions, memory, skills, tools, and agent loop directly on the device.<n>PalmClaw exposes device capabilities as device tools with explicit arguments, structured results, and clearly defined execution boundaries.<n>Experiments show an 11.5% relative improvement in task success and a 94.9% reduction in completion time over the strongest baseline.
  arXiv  Detail & Related papers  (2026-07-14T17:58:57Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence [31.954261925882452]
  This survey examines systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs.<n>We first formulate the field by the role that code plays in each task.<n>We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.
  arXiv  Detail & Related papers  (2026-06-14T17:21:43Z)
• OpenComputer: Verifiable Software Worlds for Computer-Use Agents [50.93029275079928]
  OpenComputer is a verifier-grounded framework for constructing verifiable software worlds for computer-use agents.<n>In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications.
  arXiv  Detail & Related papers  (2026-05-19T12:40:29Z)
• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback [8.080056970922483]
  Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions.<n>Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces.<n>In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments.<n>We evaluate our approach across GPT-5.4, Claude, and Qwen on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and
  arXiv  Detail & Related papers  (2026-04-14T17:55:46Z)
• AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents [18.82273534480229]
  Mobile graphical user interface (GUI) agents autonomously control applications and operating systems (OS)<n>We propose AgentRAE, a novel backdoor attack capable of inducing Remote Action Execution in mobile GUI agents using visually natural triggers.<n>Our evaluation reveals that the proposed backdoor preserves clean performance with an attack success rate of over 90% across ten mobile operations.
  arXiv  Detail & Related papers  (2026-03-24T09:51:43Z)
• WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning [51.14454312533818]
  WebGen-Agent is a novel website-generation agent that leverages comprehensive and multi-level visual feedback.<n>We introduce textitStep-GRPO with Screenshot and GUI-agent Feedback to improve the ability of LLMs to act as the reasoning engine of WebGen-Agent.<n>WebGen-Agent increases the accuracy of Claude-3.5-Sonnet from 26.4% to 51.9% and its appearance score from 3.0 to 3.9, outperforming the previous state-of-the-art agent system.
  arXiv  Detail & Related papers  (2025-09-26T17:59:51Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [57.371814877372515]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.