Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

• URL: http://arxiv.org/abs/2605.28108v2
• Date: Thu, 28 May 2026 06:48:23 GMT
• Title: Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
• Abstract summary: A long-lived LLM agent earns its value by acting on a user's preferences across sessions, not just the current request.<n>Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken.<n>As users delegate more of their affairs to agents, the impact of this gap grows.<n>ATRBench makes it measurable by fixing each user's preferences as hidden ground truth.
• Score: 18.132295539880896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.

Related papers

• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• Long-Horizon Embodied Decision-Making via Multimodal Memory Compression [71.16335993005139]
  DunphyBench is a new benchmark for evaluating agents on long-horizon human-centered embodied decision-making.<n>MeMento is a preference-conditioned multimodal memory compressor that selectively compresses decision-relevant information based on user preferences.<n>MeMento helps VLM-driven agents improve accuracy by 7.18%, while reducing memory usage by 85.38% compared to the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-02T19:29:48Z)
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions [19.069719245971786]
  We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions.<n>We curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.<n>We build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it.
  arXiv  Detail & Related papers  (2026-06-29T08:35:15Z)
• Agentic Abstention: Do Agents Know When to Stop Instead of Act? [13.780119058961331]
  LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals.<n>We define Agentic Abstention, the problem of deciding when an agent should stop acting under uncertainty.<n>We study this problem across web shopping, terminal environments, and question answering, evaluating 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks.
  arXiv  Detail & Related papers  (2026-06-27T04:49:37Z)
• Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations [14.480725554900777]
  We introduce Momento, a benchmark for persistent agentic task completion in multi-session service environments.<n> Experimental results reveal that current agents fail primarily through misestimation of user state.
  arXiv  Detail & Related papers  (2026-05-30T18:08:51Z)
• PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say [48.84133320567554]
  LLM-based agents are rapidly advancing, autonomously invoking external tools to complete tasks for users.<n> Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context.<n>We introduce emphPrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents.
  arXiv  Detail & Related papers  (2026-05-29T04:55:12Z)
• Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agents [51.46570338144688]
  ProAct is a proactive agent architecture that leverages idle-time compute to anticipate and fulfill likely upcoming user needs.<n>By analyzing evolving dialogue history together with persistent memory, ProAct predicts upcoming needs and iteratively acquires information.<n>MemBench evaluations confirm that ProAct achieves state-of-the-art reflective accuracy, underscoring its sustained and robust performance.
  arXiv  Detail & Related papers  (2026-05-25T15:47:21Z)
• Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity [12.381781997363609]
  We show that current LLM-based agents struggle to reflect or react to unexpected information.<n>We inject complete task solutions into the agent environments to deliberately expose a task's solution to a model.<n>While agents discover these solutions on Terminal-Bench in 79-81% of runs, they interact, or exploit, them in only 37-50% of cases.
  arXiv  Detail & Related papers  (2026-04-19T20:49:41Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents [4.301199871195023]
  Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering.<n>We evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified.<n>We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution.
  arXiv  Detail & Related papers  (2026-03-27T09:56:26Z)
• CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty [8.976559755301658]
  Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings.<n>We introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain.
  arXiv  Detail & Related papers  (2026-01-29T17:33:42Z)
• Are Your Agents Upward Deceivers? [73.1073084327614]
  Large Language Model (LLM)-based agents are increasingly used as autonomous subordinates that carry out tasks for users.<n>This raises the question of whether they may also engage in deception, similar to how individuals in human organizations lie to superiors to create a good image or avoid punishment.<n>We observe and define agentic upward deception, a phenomenon in which an agent facing environmental constraints conceals its failure and performs actions that were not requested without reporting.
  arXiv  Detail & Related papers  (2025-12-04T14:47:05Z)
• LLM Agents Beyond Utility: An Open-Ended Perspective [50.809163251551894]
  We augment a pretrained LLM agent with the ability to generate its own tasks, accumulate knowledge, and interact extensively with its environment.<n>It can reliably follow complex multi-step instructions, store and reuse information across runs, and propose and solve its own tasks.<n>It remains sensitive to prompt design, prone to repetitive task generation, and unable to form self-representations.
  arXiv  Detail & Related papers  (2025-10-16T10:46:54Z)
• UserBench: An Interactive Gym Environment for User-Centric Agents [110.77212949007958]
  Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, but their ability to proactively collaborate with users remains underexplored.<n>We introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions.
  arXiv  Detail & Related papers  (2025-07-29T17:34:12Z)
• Agent-Based Detection and Resolution of Incompleteness and Ambiguity in Interactions with Large Language Models [0.9856777842758593]
  This paper examines the use of agent-based architecture to bolster LLM-based Question-Answering systems with additional reasoning capabilities.<n>We equip different LLMs with agents that act as specialists in detecting and resolving deficiencies of incompleteness and ambiguity.<n>Suggesting the agent-based approach could be a useful mechanism to harness the power of LLMs to develop more robust QA systems.
  arXiv  Detail & Related papers  (2025-07-04T17:28:33Z)
• ADAPT: Actively Discovering and Adapting to Preferences for any Task [34.92667747380468]
  We introduce a benchmark designed to evaluate agents' ability to adhere to user preferences across various household tasks through active questioning.<n>Next, we propose Reflection-DPO, a novel training approach for adapting large language models to the task of active questioning.<n>We find that prior approaches that use state-of-the-art LLMs fail to sufficiently follow user preferences in ADAPT due to insufficient questioning and poor adherence to elicited preferences.
  arXiv  Detail & Related papers  (2025-04-05T03:16:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.