ProactBench: Beyond What The User Asked For

• URL: http://arxiv.org/abs/2605.09228v1
• Date: Sat, 09 May 2026 23:56:04 GMT
• Title: ProactBench: Beyond What The User Asked For
• Abstract summary: ProactBench decomposes it into three phase-tied types: textscEmergent, inference from a single disclosed anchor; textscCritical, synthesis across multiple anchors; and textscRecovery, grounded forward-looking value after task completion.<n>Our information asymmetries defend against style-confounding scoring, leakage, external-context contamination, and information dumps.
• Score: 5.422521416406412
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most LLM benchmarks score how well a model responds to explicit requests. They leave unmeasured a different conversational ability: noticing and acting on needs the user has implied but not said. We call this \emph{conversational proactivity}. ProactBench decomposes it into three phase-tied types: \textsc{Emergent}, inference from a single disclosed anchor; \textsc{Critical}, synthesis across multiple anchors; and \textsc{Recovery}, grounded forward-looking value after task completion. We operationalise the benchmark with three agents: a Planner, a User Agent, and an Assistant Model. Their information asymmetries defend against style-confounded scoring, rubric leakage, external-context contamination, and information dumps. The released corpus contains 198 curated dialogues with 624 trigger points across 24 communication styles drawn from a psychometric inventory and audited by an independent LLM judge. Across 16 frontier and open-weight models, \textsc{Recovery} is both difficult and weakly predicted by six standard benchmarks, making it a useful new evaluation signal.

Related papers

• Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion [0.0]
  Red-teaming frameworks evaluate models in multi-turn dialogues where the target model retains conversation history.<n>We introduce the textbfSAST-IR (Stateful Attacker, Stateless Target - Iterative Refinement) framework.<n>We simulate a worst-case adversarial setting using textbfmulti-turn (stateless) iterations.<n>Simple, diverse attack strategies achieved a staggering textbf96% success rate, exposing severe brittleness in memory-less defense.
  arXiv  Detail & Related papers  (2026-09-15T07:46:01Z)
• Iris: Climbing to the Search Frontier [21.066541702697112]
  We present Iris-mini and Iris-pro, two search agents trained at the 35B-A3B and 397B-A17B scales, together with the data pipeline and training recipe behind them.
  arXiv  Detail & Related papers  (2026-09-03T17:51:01Z)
• Delegation Asymmetry in Agentic Recommender Systems: Measuring Two-Sided Receptivity in Online Dating [0.0]
  We study the condition that users accept not only delegating conversation to an agent, but also receiving agent-mediated communication from others.<n>We develop a latent-threshold measurement model of agent receptivity based on graded response models with latent regression.<n>We discuss implications for agentic recommender design, including disclosure, opt-in mechanics, and receptivity-aware matchmaking.
  arXiv  Detail & Related papers  (2026-08-18T17:51:16Z)
• PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling [11.1017602726296]
  Large Language Models increasingly operate as autonomous agents.<n>Current benchmarks face three structural limitations.<n>Eval is a benchmark constructed through a two-stage framework.
  arXiv  Detail & Related papers  (2026-08-09T13:25:54Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations [32.23729177914094]
  We introduce CarryOnBench, the first interactive benchmark that measures whether LLMs can revise their interpretation of user intent and recover utility.<n>We simulate 5,970 conversations by varying user follow-up sequences, evaluating 14 models on both intent-aligned utility and safety.<n> CarryOnBench yields 1,866 different conversation flows of 4--12 turns, totaling 23,880 model responses.
  arXiv  Detail & Related papers  (2026-04-29T18:37:18Z)
• ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
  We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
  arXiv  Detail & Related papers  (2026-04-14T14:06:19Z)
• Do Chatbot LLMs Talk Too Much? The YapBench Benchmark [1.6149401958316794]
  YapBench is a benchmark for quantifying user-visible over-generation on brevity-ideal prompts.<n>Each item consists of a single-turn prompt, a curated minimal-sufficient baseline answer, and a category label.<n>We summarize model performance via the YapIndex, a uniformly weighted average of category-level median YapScores.
  arXiv  Detail & Related papers  (2026-01-02T09:43:52Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Prompt-Response Semantic Divergence Metrics for Faithfulness Hallucination and Misalignment Detection in Large Language Models [0.0]
  This paper introduces Semantic Divergence Metrics (SDM), a novel framework for detecting Faithfulness Hallucinations.<n>A heatmap of topic co-occurances between prompts and responses can be viewed as a quantified two-dimensional visualization of the user-machine dialogue.
  arXiv  Detail & Related papers  (2025-08-13T20:55:26Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• SSP: Self-Supervised Post-training for Conversational Search [63.28684982954115]
  We propose fullmodel (model) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model. To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by model on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20.
  arXiv  Detail & Related papers  (2023-07-02T13:36:36Z)
• Phrase Retrieval for Open-Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning [54.55643652781891]
  Open-Domain Conversational Question Answering (ODConvQA) aims at answering questions through a multi-turn conversation. We propose a method to directly predict answers with a phrase retrieval scheme for a sequence of words.
  arXiv  Detail & Related papers  (2023-06-07T09:46:38Z)
• Improving Open-Domain Dialogue Evaluation with a Causal Inference Model [8.625569782672663]
  Explicit satisfaction ratings can be elicited from users, but users often do not provide ratings when asked. Post-hoc ratings by experts are an alternative, but these can be both expensive and complex to collect. Here, we explore the creation of automated methods for predicting both expert and user ratings of open-domain dialogues.
  arXiv  Detail & Related papers  (2023-01-31T02:31:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.