Computer Use at the Edge of the Statistical Precipice

• URL: http://arxiv.org/abs/2605.08261v1
• Date: Thu, 07 May 2026 22:43:43 GMT
• Title: Computer Use at the Edge of the Statistical Precipice
• Abstract summary: We show that a 1MB replay script that blindly executes a recorded action sequence outperforms frontier models on prominent static benchmarks.<n>We trace failures to two root causes: non-principled environment design and non-principled evaluation methodology.
• Score: 19.604476051265852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically address. We show that a 1MB replay script that blindly executes a recorded action sequence without ever observing the screen outperforms frontier models on prominent static benchmarks, and prove that its expected success rate is exactly equal to the source agent's pass@k in deterministic environments. We trace this and other failures to two root causes: non-principled environment design (static, unsandboxed, or unreliably verified environments) and non-principled evaluation methodology (naive aggregation and misuse of pass@k for stateful UI interactions). To address the first, we propose PRISM, five design principles for CUA environments (privileged verification, realistic environments, integrity-checked configurations, sandboxed execution, and multifactorial variability) and instantiate them in DigiWorld, a benchmark of 15 realistic sandboxed mobile applications able to evaluate agents in over 3.2 million verified unique configurations. To address the second, we develop an aggregation framework pairing Wilson score intervals with hierarchical bootstrap, producing confidence intervals that correctly account for the nested structure of CUA benchmarks, as we empirically demonstrate. All together, we show that principled environment design and rigorous evaluation methodology are not optional refinements but prerequisites for meaningful CUA research.

Related papers

• EnvHarness: Awakening Static Worlds for Agent Learning [58.54600450547412]
  Environment Harness wraps a static environment to reshape its behavior without modifying the underlying logic.<n>EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier.<n>EnvRigger treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws.
  arXiv  Detail & Related papers  (2026-08-20T10:42:06Z)
• Label-Free Target-Domain Adaptation for Unconstrained Event-Image Feature Matching via Dual-Stage Distillation [30.789763956644958]
  Cross-modal matching methods are largely restricted by their reliance on either matching labels or strictly aligned hardware.<n>We propose a novel two-stage training paradigm to address this problem.<n>Our proposed method achieves state-of-the-art performance on both MVSEC and TUM-VIE pose estimation tasks.
  arXiv  Detail & Related papers  (2026-07-11T02:23:11Z)
• Hybrid Robustness Verification for Spatio-Temporal Neural Networks [19.026662893450425]
  Existing verification methods rely on overly conservative approximations or incur prohibitive computational costs.<n>In practice, adversarial perturbations exhibit structured spatial and temporal correlations, constrained to lower-dimensional, semantically meaningful subspaces.<n>We study robustness verification 3D CNNs processing video inputs, targeting applications in action recognition (UCF-101), autonomous driving (Udacity), medical imaging (MedMNIST)
  arXiv  Detail & Related papers  (2026-06-08T17:06:51Z)
• A Unified Framework for the Evaluation of LLM Agentic Capabilities [36.43241368835721]
  We present a unified framework for the fair evaluation of LLM agentic capabilities.<n> Driven by a unified configuration system, the framework integrates diverse benchmarks into a standardized instruction--tool--environment format.<n>We conduct a large-scale empirical analysis over 400K rollouts and 5B tokens on 15 models.
  arXiv  Detail & Related papers  (2026-05-27T03:20:45Z)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection [9.404376027901277]
  Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations.<n>Traditional frame-level evaluations treat video as a collection of isolated frames, fundamentally misaligned with how anomalies manifest and are acted upon in the real world.<n>We introduce two strategies for temporal event localization: a score-refinement pipeline with hierarchical Gaussian smoothing and adaptive binarization, and an end-to-end Dual-Branch Model that directly generates event-level detections.
  arXiv  Detail & Related papers  (2026-04-10T13:52:18Z)
• K$α$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks [4.297070083645049]
  We propose K$$LOS, a unified meta-algorithm that generalizes the "Localization First" principle to dataset quality evaluation.<n>By resolving spatial correspondence before assessing agreement, our framework transforms complex-categorical problems into nominal reliability.
  arXiv  Detail & Related papers  (2026-03-28T08:54:05Z)
• AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild [30.138230316314534]
  We introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction to bidirectional intent alignment.<n>We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols.<n>We also develop MUSE, an automated framework utilizing an MLLM-as-a-judge multi-agent architecture.
  arXiv  Detail & Related papers  (2026-02-12T09:25:15Z)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition [72.24180896265192]
  We introduce AgentNoiseBench, a framework for evaluating robustness of agentic models under noisy environments.<n>We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios.<n>We then categorize environmental noise into two primary types: user-noise and tool-noise.<n>Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T20:33:10Z)
• Automated Model Evaluation for Object Detection via Prediction Consistency and Reliability [3.9486037760311725]
  Prediction Consistency and Reliability (PCR) estimates detection performance without ground-truth labels.<n>We construct a meta-dataset by applying image corruptions of varying severity.<n>Results demonstrate that PCR yields more accurate performance estimates than existing AutoEval methods.
  arXiv  Detail & Related papers  (2025-08-16T15:39:56Z)
• Certifiably Robust Policies for Uncertain Parametric Environments [57.2416302384766]
  We propose a framework based on parametric Markov decision processes (MDPs) with unknown distributions over parameters.<n>We learn and analyse IMDPs for a set of unknown sample environments induced by parameters.<n>We show that our approach produces tight bounds on a policy's performance with high confidence.
  arXiv  Detail & Related papers  (2024-08-06T10:48:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.