ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm

• URL: http://arxiv.org/abs/2606.13239v1
• Date: Thu, 11 Jun 2026 11:53:32 GMT
• Title: ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm
• Abstract summary: ComActor is a self-correcting agent trained through a progressive three-stage framework.<n>ComForge is a scalable platform for large-scale training in Windows containers.<n>ComCADBench is the first benchmark for agents operating real industrial CAD software.
• Score: 44.50099842456479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing computer-use agents remain fundamentally limited in professional software manipulation: GUI-based agents suffer from fragile visual grounding and long-horizon error accumulation, while API-basedapproaches struggle with heterogeneous protocols and inaccessible commercial interfaces. In this work,we identify the Component Object Model (COM) as a unified executable abstraction, proposing COM-as-Action: a new paradigm that reframes professional software interaction as deterministic program synthesisrather than sequential visual control. To validate this paradigm in the most demanding environments, weintroduce ComCADBench, the first benchmark for agents operating real industrial CAD software. Ourexperiments reveal a substantial paradigm gap: frontier proprietary models achieve near-zero successunder GUI-based interaction, whereas COM-based execution yields substantial immediate gains. Tobridge the remaining gap between syntactic correctness and geometric accuracy, we develop ComActor, aself-correcting agent trained through a progressive three-stage framework, alongside ComForge, a scalableplatform for large-scale training in Windows containers. Extensive experiments show that ComActorachieves state-of-the-art performance on ComCADBench, with strong resilience in long-horizon taskswhere baselines collapse, and generalizes to external CAD benchmark.

Related papers

• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing [31.86100517851177]
  IterCAD is a unified multimodal agent framework for closed-loop, interactive CAD generation and editing.<n>It achieves highly competitive performance across multiple benchmarks, significantly outperforming existing approaches in both code executability and geometric precision.
  arXiv  Detail & Related papers  (2026-06-11T13:53:21Z)
• Exploring Interaction Paradigms for LLM Agents in Scientific Visualization [7.22431217973039]
  This paper examines how different types of large language model (LLM) agents perform on scientific visualization (SciVis) tasks.<n>We compare three primary interaction paradigms, including domain-specific agents with structured tool use, computer-use agents, and general-purpose coding agents.
  arXiv  Detail & Related papers  (2026-04-30T15:22:28Z)
• Step-level Optimization for Efficient Computer-use Agents [51.29573359027217]
  We argue that strong computer-use agents remain expensive and slow in practice.<n>We propose an event-driven, step-level cascade for computer-use agents.
  arXiv  Detail & Related papers  (2026-04-29T19:59:36Z)
• VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining [49.78591189918702]
  We introduce VisualToolChain-Bench (VTC-Bench) to evaluate tool-use proficiency in Multimodal Large Language Models (MLLMs)<n>To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations.<n>For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy.
  arXiv  Detail & Related papers  (2026-03-16T09:31:44Z)
• Computer-Using World Model [58.59112582915026]
  We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next user interface (UI) state.<n> CUWM first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot.<n>We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution.
  arXiv  Detail & Related papers  (2026-02-19T13:48:29Z)
• Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases [44.366968508477235]
  We introduce the Confucius Code Agent (CCA), a scalable software engineering agent that can operate at large-scales.<n>CCA is built on top of the Confucius SDK, an agent development platform structured around three complementary perspectives.<n>In addition, we introduce a meta-agent that automates the synthesis, evaluation, and refinement of agent configurations.
  arXiv  Detail & Related papers  (2025-12-11T08:05:58Z)
• Experience-Driven Exploration for Efficient API-Free AI Agents [34.38668336861503]
  KG-Agent is an experience-driven learning framework that structures an agent's raw pixel-level interactions into a persistent State-Action Knowledge Graph.<n> KG-Agent overcomes inefficient exploration by linking functionally similar but visually distinct GUI states.<n>We demonstrate significant improvements in exploration efficiency and strategic depth over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-10-17T02:53:06Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.