MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
Abstract Overview
This paper introduces a probe-based framework for evaluating vulnerability reports by replaying an agent's exploit and running executable checks of application security properties. The authors instantiate this idea as MobileCybench, a benchmark covering 13 Android applications with runnable environments, 495 hidden probes, two attack settings (malicious app and remote attacker), and two access levels (APK-only and source-visible). Unlike evaluations tied only to known vulnerabilities, the probes encode security properties, allowing them to score exploits that violate specified guarantees even if the underlying vulnerabilities were previously unknown. The benchmark is used to evaluate five coding agents and examine how attack setting and source-code visibility affect exploit success.
Novelty
The key novelty is evaluating offensive agents through executable probes that check application-specific security properties rather than relying on signatures of known vulnerabilities. This enables scoring exploits for previously unknown vulnerabilities across Android-specific attack surfaces, encompassing both on-device malicious applications and off-device remote-attacker scenarios.
Results
In APK-only evaluations, OpenCode/GPT-5.6-Sol achieved the highest malicious-app trigger rate at 53.8% and tied other agents in the remote-attacker setting at 16.7%. Across all agents and attack settings, source-visible access increased the overall trigger rate from 28.8% to 32.8%, with attribution resolving triggered configurations to 19 distinct vulnerabilities. All 124 triggered runs activated application-specific probes rather than generic probes, and benchmark development surfaced 23 previously unreported vulnerabilities.
Key Points
- MobileCybench evaluates agent-discovered exploits by replaying them and scoring outcomes with 495 executable probes over 13 Android applications.
- The benchmark covers four evaluation settings by combining two attack models—malicious app and remote attacker—with APK-only versus source-visible access.
- Empirically, agent performance improved modestly with source access, triggers were concentrated in application-specific probes, and the benchmarking effort surfaced 23 previously unreported vulnerabilities.