Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

• URL: http://arxiv.org/abs/2606.20363v1
• Date: Thu, 18 Jun 2026 15:25:42 GMT
• Title: Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
• Abstract summary: Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies.<n>We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations.<n>We present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.
• Score: 13.114768589139308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clusters are readable on the source benchmark: five of eight clusters have at least 0.95 purity against InteraSkill Workflows labels. However, readability does not imply transfer. GRPO improves IW skill-step accuracy only from 18.5\% to 20.5\%, leaves BrowseComp+ essentially unchanged, and underperforms trivial frequency priors on key source-domain metrics. We therefore present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.

Related papers

• AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories [36.7302669545557]
  Real-world applications provide no pre-defined tasks or verifiers, no faithful simulators, and limited budget for large-scale environment interaction.<n>We propose textbfAgentBrew, an offline training framework that learns effective tool-use policies from a single batch of raw interaction trajectories.<n>AgentBrew improves Qwen3-32B by +8.7 Acc / +9.7 Score on average, surpassing Qwen3-235B (+2.3 / +4.4) and outperforming rejection sampling.
  arXiv  Detail & Related papers  (2026-09-05T03:04:23Z)
• MCPShield: Content-Aware Attack Detection for LLM Agent Tool-Call Traffic [1.827510863075184]
  The Model Context Protocol (MCP) has become a widely adopted interface for agents to invoke external tools.<n> MCP is presented as an attack framework for MCP tool-call traffic that encodes each agent session as a graph.
  arXiv  Detail & Related papers  (2026-05-11T14:55:48Z)
• Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents [19.368928130932833]
  Rudder is a software module embedded in the state-of-the-art AWS DistDGL framework.<n>We introduce Rudder, a software module embedded in the state-of-the-art AWS DistDGL framework, to autonomously prefetch remote nodes.<n>We observe that the generative AI found in contemporary Large Language Models (LLMs) exhibits emergent properties like In-Context Learning (ICL) for zero-shot tasks, with logical multi-step reasoning.
  arXiv  Detail & Related papers  (2026-02-26T23:39:42Z)
• Semi-Supervised Cross-Domain Imitation Learning [9.740139304952615]
  Cross-domain imitation learning (CDIL) accelerates policy learning by transferring expert knowledge across domains.<n>We introduce the Semi-Supervised CDIL setting and propose the first algorithm for SS-CDIL.<n>Our approach achieves stable and data-efficient policy learning with minimal supervision.
  arXiv  Detail & Related papers  (2026-02-11T12:38:08Z)
• RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations [52.752467948588816]
  We propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations.<n> RSAgent queries a segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks.<n>Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance
  arXiv  Detail & Related papers  (2025-12-30T06:50:11Z)
• Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs [49.71319907864573]
  In this paper, we propose multi-agent skill discovery which enables the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector. Considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method.
  arXiv  Detail & Related papers  (2023-07-21T14:53:12Z)
• TADIL: Task-Agnostic Domain-Incremental Learning through Task-ID Inference using Transformer Nearest-Centroid Embeddings [0.0]
  We propose a novel pipeline for identifying tasks in domain-incremental learning scenarios without supervision. We leverage the lightweight computational requirements of the pipeline to devise an algorithm that decides in an online fashion when to learn a new task.
  arXiv  Detail & Related papers  (2023-06-21T00:55:02Z)
• REGTR: End-to-end Point Cloud Correspondences with Transformers [79.52112840465558]
  We conjecture that attention mechanisms can replace the role of explicit feature matching and RANSAC. We propose an end-to-end framework to directly predict the final set of correspondences. Our approach achieves state-of-the-art performance on 3DMatch and ModelNet benchmarks.
  arXiv  Detail & Related papers  (2022-03-28T06:01:00Z)
• Fewer is More: A Deep Graph Metric Learning Perspective Using Fewer Proxies [65.92826041406802]
  We propose a Proxy-based deep Graph Metric Learning approach from the perspective of graph classification. Multiple global proxies are leveraged to collectively approximate the original data points for each class. We design a novel reverse label propagation algorithm, by which the neighbor relationships are adjusted according to ground-truth labels.
  arXiv  Detail & Related papers  (2020-10-26T14:52:42Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)
• Key Points Estimation and Point Instance Segmentation Approach for Lane Detection [65.37887088194022]
  We propose a traffic line detection method called Point Instance Network (PINet) The PINet includes several stacked hourglass networks that are trained simultaneously. The PINet achieves competitive accuracy and false positive on the TuSimple and Culane datasets.
  arXiv  Detail & Related papers  (2020-02-16T15:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.