LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

• URL: http://arxiv.org/abs/2606.16802v1
• Date: Mon, 15 Jun 2026 14:42:33 GMT
• Title: LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
• Abstract summary: LabOSBench is a benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators.<n>We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels.
• Score: 46.674982266623196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment. However, directly evaluating agents on physical high-precision instruments is impractical due to high cost, safety risks, limited accessibility, and difficulty in ensuring reproducible evaluation. This motivates the need for a simulated yet realistic testbed that preserves the operational challenges of scientific instruments while enabling scalable and safe benchmarking. To this end, we introduce LabOSBench, a challenging benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators. Operating directly via a browser, LabOSBench avoids resource-heavy OS virtualization while supporting flexible task configuration and execution-based evaluation. Specifically, LabOSBench constructs 96 subtasks across eight instrument simulators, covering workflows from sample loading, alignment, parameter tuning, and data acquisition to result inspection. We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels. Our experiments reveal that while existing agents can complete many structured GUI subtasks, they still struggle with feedback-driven operations and long-horizon workflow execution. Overall, LabOSBench provides a reproducible, low-cost testbed for advancing computer-using agents toward scientific-instrument control.

Related papers

• DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? [40.322111941653134]
  We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science inside real computer environments.<n>Our experiments show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success.<n>These results reveal a substantial capability gap between current agentic systems and real data-science.
  arXiv  Detail & Related papers  (2026-08-11T01:45:56Z)
• LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories [95.20367571157679]
  Vision-Language-Action (VLA) models provide one possible interface between written protocols and robot execution.<n>Existing policies are trained mostly on household and tabletop demonstrations and rarely encounter the instruments, transparent liquids, or fixed protocol found in scientific laboratories.<n>We present LabVLA, trained with a two-stage recipe: FAST action token pretraining first makes the Qwen3-VL-4B-Instruct backbone action aware before any continuous control is learned, and flow matching posttraining then attaches a DiT action expert under knowledge insulation.
  arXiv  Detail & Related papers  (2026-06-11T17:03:53Z)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? [29.5539656241093]
  Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments.<n>Existing benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions.<n>We introduce a benchmark built on 23 deployable systems across six professional domains, containing 106 tasks grounded in realistic work scenarios.<n>Experiments show that representative LLM-based agents struggle on-Bench, with even the strongest model completing fewer than 4% of tasks end-to-end.
  arXiv  Detail & Related papers  (2026-05-15T09:35:15Z)
• Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants [85.1155076383488]
  Pare is a framework for building and evaluating proactive agents in digital environments.<n>Pare-Bench is a benchmark of 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps.
  arXiv  Detail & Related papers  (2026-04-01T12:53:01Z)
• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• Computer-Using World Model [58.59112582915026]
  We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next user interface (UI) state.<n> CUWM first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot.<n>We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution.
  arXiv  Detail & Related papers  (2026-02-19T13:48:29Z)
• See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch [20.231957791642635]
  We introduce ScratchWorld, a benchmark for evaluating multimodal GUI agents on program-by-construction tasks in Scratch.<n>ScratchWorld comprises 83 curated tasks spanning four distinct problem categories: Create, Debug, Extend, and Compute.<n>To ensure reliable assessment, we propose an execution-based evaluation protocol that validates the functional correctness of the constructed Scratch programs.
  arXiv  Detail & Related papers  (2026-02-11T12:54:53Z)
• AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act [27.922630781100864]
  We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation.<n>We benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments.<n>Our benchmark reveals several failure modes that offline vision-language tests fail to capture.
  arXiv  Detail & Related papers  (2026-02-02T05:30:14Z)
• LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents [103.65422553044816]
  LabUtopia is a comprehensive simulation and benchmarking suite designed to facilitate the development of generalizable, reasoning-capable embodied agents.<n>It supports 30 distinct tasks and includes more than 200 scene and instrument assets.<n>We demonstrate that LabUtopia offers a powerful platform for advancing the integration of perception, planning, and control in scientific-purpose agents.
  arXiv  Detail & Related papers  (2025-05-28T17:50:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.