From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents

• URL: http://arxiv.org/abs/2609.04869v1
• Date: Fri, 04 Sep 2026 08:29:45 GMT
• Title: From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents
• Abstract summary: We present an online skill-evolution framework that converts interaction trajectories and evaluator feedback into a persistent, versioned library of reusable procedures.<n>We compare the full evolving-library system with a configuration-matched empty-library control across four OSWorld application domains.<n>In GIMP, provenance-aware analysis reveals retrieval across task-of-origin boundaries and revision, where repeated accepted edits fail to recover the originating task.
• Score: 48.50635499112162
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer-use agents can execute increasingly complex tasks in graphical interfaces, but their interaction experience is typically transient: procedural knowledge acquired from one rollout is not systematically retained, refined, and reused in later tasks. Existing skill libraries provide external procedural knowledge, yet their incremental value over the same agent operating without skills, as well as their longitudinal dynamics under repeated interaction, remain insufficiently characterized. We present an online skill-evolution framework that converts interaction trajectories and evaluator feedback into a persistent, versioned library of reusable procedures. Each iteration executes against a frozen library snapshot, and evidence-guided skill updates become available in subsequent iterations without changing model parameters. We compare the full evolving-library system with a configuration-matched empty-library control across four OSWorld application domains under the same fixed action-generation and GUI-grounding stack, task sets, and iteration horizons. Following a five-iteration empty-library warm-up, Full attains a higher post-warm-up mean evaluator score in all four observed domain runs, with mean differences ranging from 5.7 to 18.6 percentage points and domain-dependent temporal stability. In GIMP, provenance-aware analysis reveals retrieval across task-of-origin boundaries and revision churn, where repeated accepted edits fail to recover the originating task. These findings characterize evolving skill libraries as auditable, shared procedural memory that can improve a fixed computer-use stack, while showing that their benefits are conditional and repeated revision does not guarantee recovery. Code is released at https://github.com/LongtaoHu/Skill-Evo4GUI.

Related papers

• SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams [55.90739575113059]
  We argue the missing unit of reuse is the solving procedure shared by a cluster of related tasks, and build SkillGLoW around it.<n>Across four benchmarks (mathematical reasoning, terminal automation, software repair, and embodied control) and three models, the priors gain 17.2 points (hard) over the no-skill baseline on average.
  arXiv  Detail & Related papers  (2026-09-02T07:31:18Z)
• SCAFFOLD: Self-Improving Web Agents via Recursive Parametric Skill Abstraction [13.7368753473801]
  textscScaffold is a self-improving framework for visual web agents.<n>It induces parametric, executable skills from successful trajectories under a multi-instance abstraction constraint.<n>textscScaffold improves success rate by $11.1$--$17.2$ absolute points over the strongest skill-augmented baseline.
  arXiv  Detail & Related papers  (2026-08-31T08:16:56Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• From Cognitive Architectures to Language Agents: A Mechanism-Level Review of Lineage, Convergence, and Migration Gaps [0.0]
  Review connects ten historical cognitive architectures, eight language-agent runtime families, and forty-two mechanism-focused modern systems.<n>We reconstruct each mechanism through state, control, transition, persistence, failure, learning, and resource governance.<n>We contribute a distinctive-mechanism catalog, an auditable evidence-depth framework, and a falsifiable agenda for testing these bundles as composable invariants.
  arXiv  Detail & Related papers  (2026-07-27T02:38:59Z)
• Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns [68.0603867264595]
  SkillMigrator learns reusable web skills and transfers them across sites by matching layout structure rather than specific element references.<n>It reduces the average LLM-action count on successful trajectories by 8-10% across both WebArena and Mind2Web at matched success rate.
  arXiv  Detail & Related papers  (2026-06-16T08:04:35Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills [31.23929961213889]
  Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks.<n>We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation.<n>We find that current agents often adapt locally but rarely form robust reusable skills.
  arXiv  Detail & Related papers  (2026-05-22T18:23:31Z)
• REVERE: Reflective Evolving Research Engineer for Scientific Workflows [44.36828076189902]
  Existing prompt-optimization techniques rely on local signals to update behavior, often broader and recurring patterns across tasks, leading to poor generalization.<n>We introduce Reflective Evolving Research Engineer (REVERE), a framework that continuously learns from Global Training Context.<n>REVERE, via this reflective optimization framework, improves performance over prior state-of-the-art expert-crafted instructions on research coding tasks by 4.50% on SUPER, 3.51% on ResearchCodeBench, and 4.89% on ScienceAgentBench.
  arXiv  Detail & Related papers  (2026-03-21T05:58:30Z)
• AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agents [26.380991138110925]
  AutoAgent is a self-evolving multi-agent framework built on evolving cognition, on-the-fly contextual decision-making, and elastic memory orchestration.<n>Each agent maintains structured prompt-level cognition over tools, self-capabilities, peer expertise, and task knowledge.<n>AutoAgent consistently improves task success, tool-use efficiency, and collaborative robustness over static and memory-augmented baselines.
  arXiv  Detail & Related papers  (2026-03-10T14:23:49Z)
• SWE-Bench-CL: Continual Learning for Coding Agents [0.0]
  SWE-Bench-CL is a novel continual learning benchmark built on the human-verified SWE-Bench Verified dataset.<n>By organizing GitHub issues into chronologically ordered sequences that reflect natural repository evolution, SWE-Bench-CL enables direct evaluation of an agent's ability to accumulate experience.
  arXiv  Detail & Related papers  (2025-06-13T07:11:14Z)
• CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization [62.0397906276669]
  CLIN is the first language-based agent to continually improve over multiple trials. It can improve its zero-shot performance by 4 points (13 for new tasks) and can further improve performance there through continual memory updates. This suggests a new architecture for agents built on frozen models that can still continually and rapidly improve over time.
  arXiv  Detail & Related papers  (2023-10-16T07:17:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.