PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions

• URL: http://arxiv.org/abs/2606.14832v1
• Date: Fri, 12 Jun 2026 15:01:32 GMT
• Title: PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions
• Abstract summary: PhoneHarness is a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile.<n>Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers.<n>Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
• Score: 73.69976712292683
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.

Related papers

• EchoPath: Execution-Level Replayable Memory for GUI Agents [45.101663136838916]
  We introduce EchoPath, a model-agnostic harness that converts artifact-validated GUI trajectories into standardized, parameter-controlled callable memories.<n>Each memory stores task-intent keys, application and state preconditions, flexible input parameters, GUI evidence, validation provenance, and lifecycle state.<n>In experiments with real computer-use tasks, EchoPath reduced median token cost by more than 90% and median execution time by about 60%.
  arXiv  Detail & Related papers  (2026-09-15T04:56:17Z)
• MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks [37.73689537216401]
  We present textbfMobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents.<n>MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools.<n>By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning.
  arXiv  Detail & Related papers  (2026-08-24T09:38:24Z)
• PalmClaw: A Native On-Device Agent Framework for Mobile Phones [16.173016265644673]
  We present textbfPalmClaw, an open-source agent framework that runs on mobile phones and manages the sessions, memory, skills, tools, and agent loop directly on the device.<n>PalmClaw exposes device capabilities as device tools with explicit arguments, structured results, and clearly defined execution boundaries.<n>Experiments show an 11.5% relative improvement in task success and a 94.9% reduction in completion time over the strongest baseline.
  arXiv  Detail & Related papers  (2026-07-14T17:58:57Z)
• A Task-State Representation for Long-Horizon Mobile GUI Agents [53.359524744953575]
  Task-State Representation (TSR) is a training-free framework that explicitly decouples task state from sensory input.<n> Experiments across four mobile GUI benchmarks validate TSR's effectiveness, yielding up to a 12 absolute point increase in success rate.
  arXiv  Detail & Related papers  (2026-07-01T06:37:21Z)
• PhoneBuddy: Training Open Models for Agentic Phone Use [104.23243994660936]
  PhoneBuddy is a training recipe and open-model line for agentic phone use.<n>It combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs run mock apps from real GUI usage structure.<n>PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL.
  arXiv  Detail & Related papers  (2026-06-22T08:57:54Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• Computer-Use Agents as Judges for Generative User Interface [142.75272102498806]
  ComputerUse Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI)<n>Most GUI remain designed primarily for humans to adopt human-oriented behaviors that are unnecessary for efficient task execution.<n>This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design?
  arXiv  Detail & Related papers  (2025-11-19T16:00:02Z)
• GTA1: GUI Test-time Scaling Agent [97.58177633084915]
  Graphical user interface (GUI) agents autonomously complete tasks across platforms (eg, Linux) by sequentially decomposing user instructions into action proposals.<n>This paper investigates the aforementioned challenges with our textbfGUI textbfTest-time Scaling textbfAgent, namely GTA1.
  arXiv  Detail & Related papers  (2025-07-08T08:52:18Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• MAPLE: A Mobile Agent with Persistent Finite State Machines for Structured Task Reasoning [46.18718721121415]
  We present MAPLE, a state-aware multi-agent framework that abstracts app interactions as a Finite State Machine (FSM)<n>We computationally model each UI screen as a discrete state and user actions as transitions, allowing the FSM to provide a structured representation of the app execution.<n> MAPLE consists of specialized agents responsible for four phases of task execution: planning, execution, verification, error recovery, and knowledge retention.
  arXiv  Detail & Related papers  (2025-05-29T16:08:51Z)
• LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation [8.998467488526327]
  This paper presents LlamaTouch, a testbed for on-device mobile UI task execution and faithful, scalable task evaluation. LlamaTouch employs a novel evaluation approach that only assesses whether an agent traverses all manually annotated, essential application/system states. LlamaTouch also enables easy task annotation and integration of new mobile agents.
  arXiv  Detail & Related papers  (2024-04-12T15:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.