OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks

• URL: http://arxiv.org/abs/2601.20650v2
• Date: Mon, 02 Feb 2026 12:28:52 GMT
• Title: OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks
• Abstract summary: Long-horizon, repetitive tasks are common in professional settings.<n>These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process.<n>We establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents.
• Score: 36.99798674847767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon, repetitive workflows are common in professional settings, such as processing expense reports from receipts and entering student grades from exam papers. These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process. However, they are ideal for Computer-Use Agents (CUAs) due to their structured, recurring sub-workflows with logic that can be systematically learned. Identifying the absence of an evaluation benchmark as a primary bottleneck, we establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents. We then introduce a cost-effective method to construct a condensed demonstration using only few-shot examples to teach agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections. Extensive experiments demonstrate both the inherent challenges of these tasks and the effectiveness of our proposed method. Project website: https://os-marathon.github.io/.

Related papers

• FlowScout: From Execution Feedback to Reliable Tool-Using Agent Workflows [6.767792769288717]
  FlowScout is an execution-guided framework for generating tool-integrated agentic from historical task-solving records.<n>It first mines a common tool coordination from historical records to construct an initial workflow.<n>It then refines the workflow topology through Monte Carlo tree search guided by execution feedback.
  arXiv  Detail & Related papers  (2026-08-10T08:41:59Z)
• From Search to Synthesis: Training LLMs as Zero-Shot Workflow Generators [17.263246226058474]
  Large language models (LLMs) excel across a wide range of tasks, yet their instance-specific solutions often lack the structural consistency needed for reliable deployment.<n>We present MetaFlow, which casts workflow generation as a meta-learning problem.<n>It trains in two stages: supervised fine-tuning on synthetic workflow data, followed by reinforcement learning with verifiable rewards.<n>The resulting model produces effective for trained tasks and exhibits strong generalization to untrained tasks and novel operator sets.
  arXiv  Detail & Related papers  (2026-06-29T14:17:09Z)
• SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding [41.98672557723593]
  We introduce SWEQA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments.<n>We also propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF)<n> Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art models.
  arXiv  Detail & Related papers  (2026-03-17T05:12:48Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• Do We Always Need Query-Level Workflows? Rethinking Agentic Workflow Generation for Multi-Agent Systems [72.3575737073235]
  Multi-Agent Systems (MAS) solve complex tasks by coordinating multiple agents through.<n>Existing approaches generates either at task level or query level, but their relative costs and benefits remain unclear.<n>We show that query-level workflow generation is not always necessary, since a small set of top-K best task-level together already covers equivalent or even more queries.
  arXiv  Detail & Related papers  (2026-01-16T10:05:51Z)
• Beyond Rule-Based Workflows: An Information-Flow-Orchestrated Multi-Agents Paradigm via Agent-to-Agent Communication from CORAL [0.15199492741752027]
  We propose an Information-Flow-Orchestrated Multi-Agent Paradigm via Agent-to-Agent (A2A) Communication.<n>We evaluate our approach on the general-purpose benchmark GAIA, using the representative workflow-based MAS as the baseline.<n>Our method achieves 63.64% accuracy, outperforming OWL's 55.15% by 8.49 percentage points with comparable token consumption.
  arXiv  Detail & Related papers  (2026-01-14T21:35:51Z)
• Solving Context Window Overflow in AI Agents [0.0]
  Large Language Models (LLMs) have become increasingly capable of interacting with external tools, granting access to specialized knowledge beyond their training data.<n>Existing solutions such as truncation or summarization fail to preserve complete outputs, making them unsuitable for work requiring the full data.<n>This paper introduces a method that enables LLMs to process and utilize tool responses of arbitrary length without loss of information.
  arXiv  Detail & Related papers  (2025-11-27T19:22:20Z)
• Residual Multi-Task Learner for Applied Ranking [11.774841918446137]
  ResFlow is a lightweight multi-task learning framework that enables efficient cross-task information sharing. It is fully deployed in the pre-rank module of Shopee Search.
  arXiv  Detail & Related papers  (2024-10-30T06:49:45Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Agent Workflow Memory [71.81385627556398]
  We introduce Agent Memory, a method for inducing commonly reused routines. AWM substantially improves the baseline results by 24.6% and 51.1% relative success rate. Online AWM robustly generalizes in cross-task, website, and domain evaluations.
  arXiv  Detail & Related papers  (2024-09-11T17:21:00Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.