Gym-Anything: Turn any Software into an Agent Environment

• URL: http://arxiv.org/abs/2604.06126v1
• Date: Tue, 07 Apr 2026 17:38:15 GMT
• Title: Gym-Anything: Turn any Software into an Agent Environment
• Abstract summary: Gym-Anything is a framework for converting any software into an interactive computer-use environment.<n> CUA-World is a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems.
• Score: 67.2443447990221
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-language model outperforms models 2$\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.

Related papers

• DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? [40.322111941653134]
  We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science inside real computer environments.<n>Our experiments show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success.<n>These results reveal a substantial capability gap between current agentic systems and real data-science.
  arXiv  Detail & Related papers  (2026-08-11T01:45:56Z)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? [29.5539656241093]
  Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments.<n>Existing benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions.<n>We introduce a benchmark built on 23 deployable systems across six professional domains, containing 106 tasks grounded in realistic work scenarios.<n>Experiments show that representative LLM-based agents struggle on-Bench, with even the strongest model completing fewer than 4% of tasks end-to-end.
  arXiv  Detail & Related papers  (2026-05-15T09:35:15Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution [86.4588675093384]
  Toolathlon is a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation.<n>This benchmark includes 108 manually sourced or crafted tasks, requiring interacting with multiple Apps over around 20 turns on average to complete.<n>We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.
  arXiv  Detail & Related papers  (2025-10-29T17:32:49Z)
• Programming with Pixels: Can Computer-Use Agents do Software Engineering? [24.011063667060792]
  $textttProgramming with Pixels$ (PwP) is the first comprehensive computer-use environment for software engineering.<n>PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance.
  arXiv  Detail & Related papers  (2025-02-24T18:41:33Z)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments [87.41051677852231]
  We introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks. We create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and spanning multiple applications.
  arXiv  Detail & Related papers  (2024-04-11T17:56:05Z)
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study [72.24266814625685]
  We explore the performance of large language models (LLMs) across the entire software development lifecycle with DevEval.<n>DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task.<n> Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval.
  arXiv  Detail & Related papers  (2024-03-13T15:13:44Z)
• Towards Lightweight Data Integration using Multi-workflow Provenance and Data Observability [0.2517763905487249]
  Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era. We propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.
  arXiv  Detail & Related papers  (2023-08-17T14:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.