Syll: Open-Source Personal Automation with Cross-Surface Execution

• URL: http://arxiv.org/abs/2606.07594v1
• Date: Thu, 28 May 2026 17:59:31 GMT
• Title: Syll: Open-Source Personal Automation with Cross-Surface Execution
• Abstract summary: We present Syll, an open-source, self-hosted multimodal agent harness.<n>Syll unifies MCP/API tools, CLI execution, and visual control in a modular runtime.<n>Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, Finder.
• Score: 73.77463150026068
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface and offer limited support for user teaching and auditability. We present Syll, an open-source, self-hosted multimodal agent harness that unifies MCP/API tools, CLI execution, and visual GUI control in a modular runtime, enabling agents to coordinate computer use across heterogeneous interfaces while streamlining how users and agents exchange information. At the core of Syll is a bidirectional user-agent interaction layer: users teach procedures through direct demonstration, which Syll compiles into reusable skills; agent execution is translated back into multimodal evidence -- logs, keyframes, and approval checkpoints -- for inspection and control. Syll further externalizes memory, skills, routines, and governance as editable local artifacts, supporting straightforward inspection, extension, and downstream development. Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, macOS Finder and others. We report mechanism-oriented studies that validate multimodal routing, teachable GUI replay, and persistent local artifacts. We hope Syll can serve as a practical open-source foundation for personal automation that users can teach, inspect, and continuously extend.

Related papers

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition [51.15720718264743]
  We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device requiring coordinated interaction across platforms.<n>JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device transformations.<n>We show that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world usage.
  arXiv  Detail & Related papers  (2026-09-09T16:59:59Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• AppAgent-Claw: CLI Is All You Need for GUI Automation [9.116481317329493]
  AppAgent-Claw is a demonstration-driven system that converts GUI into reliable, reusable skills without runtime inference.<n>It employs a layered localization strategy to handle visual shifts and a validation-coupled execution model to ensure intended on-screen effects.
  arXiv  Detail & Related papers  (2026-04-15T17:02:09Z)
• AgentSPEX: An Agent SPecification and EXecution Language [17.439531520677594]
  We introduce AgentSPEX, an agent framework with explicit control flow and modular structure.<n>AgentSPEX supports typed steps, branching and loops, parallel execution, reusable submodules, and explicit state management.<n>We show through a user study that AgentSPEX provides a more interpretable and accessible workflow-authoring paradigm than a popular existing agent framework.
  arXiv  Detail & Related papers  (2026-04-14T23:16:25Z)
• AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem [39.8098595660156]
  This paper proposes a new paradigm: a Personal Agent Operating System (AgentOS)<n>In AgentOS, traditional GUI desktops are replaced by a Natural User Interface (NUI) centered on a unified natural language or voice portal.<n>The system core becomes an Agent Kernel that interprets user intent, decomposes tasks, and coordinates multiple agents.
  arXiv  Detail & Related papers  (2026-03-09T21:13:52Z)
• Computer-Use Agents as Judges for Generative User Interface [142.75272102498806]
  ComputerUse Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI)<n>Most GUI remain designed primarily for humans to adopt human-oriented behaviors that are unnecessary for efficient task execution.<n>This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design?
  arXiv  Detail & Related papers  (2025-11-19T16:00:02Z)
• GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes [58.197090145723735]
  We introduce a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes.<n>GuiSpector extracts detailed NL feedback from the agent's verification process, providing developers with actionable insights.<n>We present an integrated tool that unifies these capabilities, offering an interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process.
  arXiv  Detail & Related papers  (2025-10-06T13:15:24Z)
• UFO2: The Desktop AgentOS [60.317812905300336]
  UFO2 is a multiagent AgentOS for Windows desktops that elevates into practical, system-level automation.<n>We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs.<n>Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation.
  arXiv  Detail & Related papers  (2025-04-20T13:04:43Z)
• AppAgent v2: Advanced Agent for Flexible Mobile Interactions [57.98933460388985]
  This work introduces a novel LLM-based multimodal agent framework for mobile devices.<n>Our agent constructs a flexible action space that enhances adaptability across various applications.<n>Our results demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios.
  arXiv  Detail & Related papers  (2024-08-05T06:31:39Z)
• ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems [39.85101344037394]
  ReDel is a toolkit for building multi-agent systems. It supports custom tool-use, delegation schemes, event-based logging, and interactive replay. Our code, documentation, and PyPI package are open-source and free to use under the MIT license.
  arXiv  Detail & Related papers  (2024-08-05T05:43:23Z)
• You Only Look at Screens: Multimodal Chain-of-Action Agents [37.118034745972956]
  Auto-GUI is a multimodal solution that directly interacts with the interface. We propose a chain-of-action technique to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions.
  arXiv  Detail & Related papers  (2023-09-20T16:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.