StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

• URL: http://arxiv.org/abs/2607.22798v1
• Date: Fri, 24 Jul 2026 14:17:03 GMT
• Title: StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
• Abstract summary: StateAct is a code-first, multi-agent harness built around this distinction.<n>Its main agent works directly with program state by using code.<n>A dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it.
• Score: 55.650871465979066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.

Related papers

• SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams [55.90739575113059]
  We argue the missing unit of reuse is the solving procedure shared by a cluster of related tasks, and build SkillGLoW around it.<n>Across four benchmarks (mathematical reasoning, terminal automation, software repair, and embodied control) and three models, the priors gain 17.2 points (hard) over the no-skill baseline on average.
  arXiv  Detail & Related papers  (2026-09-02T07:31:18Z)
• Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents [48.967927222079965]
  Ledger is a runtime layer that distills an agent's completed interactions into an explicit execution state.<n> Ledger keeps this state in an online execution ledger and applies it at two boundaries of every step.<n>Across all 500 SWE-bench Verified instances, Ledger raises Pass@1 from 56.2% to 64.2% with GPT-5 mini and from 75.8% to 81.0% with MiniMax M2.5.
  arXiv  Detail & Related papers  (2026-08-01T18:25:00Z)
• Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? [3.9561795512466413]
  Current benchmarks primarily measure end-task success or single-frame grounding.<n>This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous.<n>We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories.
  arXiv  Detail & Related papers  (2026-07-28T17:49:51Z)
• Tactile: Giving Computer-Using Agents Hands and Feet [9.655595397922488]
  We present Tactile, an open-source tool layer that gives agents a more reliable "hands and feet" for desktop use.<n>Tactile converts heterogeneous UI evidence-adapted-operating-system accessibility semantics, OCR-grounded text, and visual fallback regions--into action-grounded interface states.<n>Agents operate through an observe-ground-act-verify loop that prefers native semantic actions when available, falls back to OCR-grounded coordinates when visible text is the best evidence, and keeps full provenance for replay and failure attribution.
  arXiv  Detail & Related papers  (2026-07-16T00:35:25Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens [65.53494487819053]
  Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services.<n>We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role.<n>We develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces.
  arXiv  Detail & Related papers  (2026-06-29T18:00:45Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation [11.272830796781925]
  We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks.<n>Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass.<n>We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:00:57Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• The Amazing Agent Race: Strong Tool Users, Weak Navigators [27.24330141815882]
  Existing tool-use benchmarks for LLM agents are overwhelmingly linear.<n>We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph puzzles (or "legs") with fork-merge tool chains.<n>We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs)
  arXiv  Detail & Related papers  (2026-04-11T15:58:29Z)
• Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents [9.457255218406333]
  Computer-using agents (CUAs) act directly on graphical user interfaces, yet their perception of the screen is often unreliable.<n>We formalize the visual confused deputy: a failure mode in which an agent authorizes an action based on a misperceived screen state.<n>To mitigate this threat, we propose the first guardrail that operates outside the agent's perceptual loop.
  arXiv  Detail & Related papers  (2026-03-16T01:31:32Z)
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos [62.29924199978745]
  VideoAgentTrek is a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale.<n>Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels.<n>applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically.
  arXiv  Detail & Related papers  (2025-10-22T11:25:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.