VISUALSKILL: Multimodal Skills for Computer-Use Agents

• URL: http://arxiv.org/abs/2606.18448v1
• Date: Tue, 16 Jun 2026 19:57:07 GMT
• Title: VISUALSKILL: Multimodal Skills for Computer-Use Agents
• Abstract summary: We propose VISUALSKILL: a hierarchical multimodal skill tailored to each target application.<n>We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration.<n>On two CUA benchmarks, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL.
• Score: 63.79539541125141
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents (CUAs) approach human-level performance on standardised benchmarks but still struggle on long-horizon tasks and unseen software. Existing skill libraries address this with reusable skills, but represent the skill artifact as text only, despite the visual nature of GUI interaction. We propose VISUALSKILL: a hierarchical multimodal skill, tailored to each target application and organised as a central index over per-topic files, which the agent consumes through a load_topic MCP tool that fetches the relevant topic's text and figures on demand. We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration. On two CUA benchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL, a +15.3 point absolute lift over the no-skill baseline (0.303). Against a matched text-only skill that is generated from the same source content and differs from VISUALSKILL only in modality, VISUALSKILL yields a further +8.3 point absolute gain over the matched text-only skill (0.373 vs. 0.456), providing direct evidence that retaining visual figures in the skill artifact, rather than verbalizing them away, helps the agent both identify UI elements and verify workflow state after each action. Our code is available at https://github.com/XMHZZ2018/VisualSkills.

Related papers

• UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp [14.339105360523831]
  Real-world multimodal browsing involves three distinct information-flow patterns: text-only, image-to-text, and text-to-image.<n>We introduce UNIBROWSE, a unified data pipeline that simultaneously generates training data covering all three patterns.<n>The resulting UNIBROWSE agent achieves state-of-the-art performance on multimodal BrowseComp benchmarks.
  arXiv  Detail & Related papers  (2026-07-12T04:13:27Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns [68.0603867264595]
  SkillMigrator learns reusable web skills and transfers them across sites by matching layout structure rather than specific element references.<n>It reduces the average LLM-action count on successful trajectories by 8-10% across both WebArena and Mind2Web at matched success rate.
  arXiv  Detail & Related papers  (2026-06-16T08:04:35Z)
• VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents [25.141059096863255]
  VISTA is a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents.<n>We define five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint.<n>Each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points.
  arXiv  Detail & Related papers  (2026-05-22T20:29:12Z)
• SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs [70.1970574147839]
  We propose SKILLGRAPH, a framework that represents reusable skills as nodes in a directed graph.<n>SKILLGRAPH retrieves not just individual skills, but an ordered skill subgraph that can guide multi-step decision making.<n> Experiments show that SKILLGRAPH achieves state-of-the-art performance against memory-augmented RL methods.
  arXiv  Detail & Related papers  (2026-05-12T12:21:49Z)
• SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills [0.0]
  SkillSieve is a three-layer detection framework that applies progressively deeper analysis only where needed.<n>On a 400-skill labeled benchmark, SkillSieve achieves 0.800 F1, outperforming ClawVet's 0.421, at an average cost of 0.006 per skill.
  arXiv  Detail & Related papers  (2026-04-08T00:58:48Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale [28.43462779191672]
  AgentSkillOS is a principled framework for skill selection, orchestration, and ecosystem-level management.<n>AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree.<n> (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines.
  arXiv  Detail & Related papers  (2026-03-02T18:46:47Z)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents [49.34040731113563]
  We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities.<n> Rigorous manual validation yields 158 high-quality tools, each verified for correct functionality, practical applicability, and versatility.<n> OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments.
  arXiv  Detail & Related papers  (2025-10-28T15:56:36Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• See and Think: Embodied Agent in Virtual Environment [12.801720916220823]
  Large language models (LLMs) have achieved impressive pro-gress on several open-world tasks. This paper proposes STEVE, a comprehensive and visionary embodied agent in the Minecraft virtual environment.
  arXiv  Detail & Related papers  (2023-11-26T06:38:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.