KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation

• URL: http://arxiv.org/abs/2604.08455v1
• Date: Thu, 09 Apr 2026 16:50:50 GMT
• Title: KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
• Abstract summary: KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
• Score: 72.01173512175531
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.

Related papers

• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• Personalizing Large Language Model Agents with Small Policy Models [6.759195695948836]
  FABLE is a lightweight policy layer outside a potentially black-box host agent.<n>It factorizes memory, information-acquisition, and response decisions so feedback updates related choices.<n>It learns user-specific residual preferences relative to a fixed default-and-cost score.
  arXiv  Detail & Related papers  (2026-07-31T18:56:30Z)
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions [19.069719245971786]
  We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions.<n>We curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.<n>We build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it.
  arXiv  Detail & Related papers  (2026-06-29T08:35:15Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions [63.13827503828231]
  We introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions.<n>Results show that real-world personalization remains highly challenging even for state-of-the-art models.
  arXiv  Detail & Related papers  (2026-05-26T15:07:38Z)
• Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent [22.126763421836966]
  We introduce SpecBench, a new benchmark for evaluating an agent's ability to translate user intent into a structured, executable specification.<n>The agent is given access to past user conversations and may interact with the user for a fixed number of rounds to ask clarifying questions.
  arXiv  Detail & Related papers  (2026-05-13T16:07:43Z)
• AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild [30.138230316314534]
  We introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction to bidirectional intent alignment.<n>We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols.<n>We also develop MUSE, an automated framework utilizing an MLLM-as-a-judge multi-agent architecture.
  arXiv  Detail & Related papers  (2026-02-12T09:25:15Z)
• UserBench: An Interactive Gym Environment for User-Centric Agents [110.77212949007958]
  Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, but their ability to proactively collaborate with users remains underexplored.<n>We introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions.
  arXiv  Detail & Related papers  (2025-07-29T17:34:12Z)
• PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time [87.99027488664282]
  PersonaAgent is a framework designed to address versatile personalization tasks.<n>It integrates a personalized memory module and a personalized action module.<n>Test-time user-preference alignment strategy ensures real-time user preference alignment.
  arXiv  Detail & Related papers  (2025-06-06T17:29:49Z)
• Creating General User Models from Computer Use [53.59999173952482]
  This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer.<n>The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences.
  arXiv  Detail & Related papers  (2025-05-16T04:00:31Z)
• Aligning LLM Agents by Learning Latent Preference from User Edits [23.235995078727658]
  We study interactive learning of language agents based on user edits made to the agent's output. We propose a learning framework, PRELUDE, that infers a description of the user's latent preference based on historic edit data. We introduce two interactive environments -- summarization and email writing, and use a GPT-4 simulated user for evaluation.
  arXiv  Detail & Related papers  (2024-04-23T17:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.