InteractComp: Evaluating Search Agents With Ambiguous Queries

• URL: http://arxiv.org/abs/2510.24668v1
• Date: Tue, 28 Oct 2025 17:35:54 GMT
• Title: InteractComp: Evaluating Search Agents With Ambiguous Queries
• Abstract summary: We introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search.<n> Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context.<n>This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents.
• Score: 36.05005463045869
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language agents have demonstrated remarkable potential in web search and information retrieval. However, these search agents assume user queries are complete and unambiguous, an assumption that diverges from reality where users begin with incomplete queries requiring clarification through interaction. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates genuine ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp.

Related papers

• Mitigating Context Interference for Reliable and Efficient Search Agents [67.82705889007238]
  This study investigates context interference in multi-turn search agents.<n>We introduce a distill-based context refiner to dynamically mitigate context interference for multi-turn search agents.<n>We validate that incorporating context refinement into RL training pipelines of search agents can significantly enhance both reliability and efficiency.
  arXiv  Detail & Related papers  (2026-08-11T10:00:42Z)
• SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task [28.244711705405056]
  Training effective search agents remains challenging due to the lack of scalable and long-horizon tasks.<n>We introduce SearchArt, a framework for training long-horizon search agents.<n>SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks.
  arXiv  Detail & Related papers  (2026-07-25T06:47:46Z)
• When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search [6.384441153713447]
  We introduce DiscoBench, a benchmark for clarification-aware deep search.<n>It is designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction.<n>Experiments show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing.
  arXiv  Detail & Related papers  (2026-06-26T02:57:15Z)
• VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild [0.0]
  LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying.<n>We attribute this gap to existing benchmarks' reliance on over-specified queries, single-turn interactions, and fixed-schema evaluation.<n>We term this paradigm VibeSearch and introduce VibeSearchBench, a benchmark comprising 200 manually curated bilingual (Chinese and English) tasks across 20 domains.
  arXiv  Detail & Related papers  (2026-05-27T03:06:18Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• AgentSearchBench: A Benchmark for AI Agent Search in the Wild [21.739444037726418]
  We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild.<n>The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions.<n>We show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality.
  arXiv  Detail & Related papers  (2026-04-24T10:53:54Z)
• AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learning [52.305422887002656]
  We propose a reinforcement learning framework that evaluates each search step via self-generated intermediate answers.<n>By a self-answering mechanism, AutoSearch identifies the minimal sufficient search depth and promotes efficient search.<n>Experiments show that AutoSearch achieves a superior accuracy-efficiency trade-off, alleviating over-searching while preserving search quality.
  arXiv  Detail & Related papers  (2026-04-19T09:05:48Z)
• Rethinking Deep Research from the Perspective of Web Content Distribution Matching [86.90255285277256]
  We propose WeDas, a Web Content Distribution Aware framework that incorporates search-space structural characteristics into the agent's observation space.<n>We introduce a few-shot probing mechanism that iteratively estimates the Query-Result Alignment Score.<n>As a plug-and-play module, WeDas consistently improves sub-goal completion and accuracy across four benchmarks.
  arXiv  Detail & Related papers  (2026-03-07T14:51:17Z)
• PATHWAYS: Evaluating Investigation and Context Discovery in AI Web Agents [0.0]
  PATHWAYS is a benchmark of 250 multi-step decision tasks.<n>It tests whether web-based agents can discover and correctly use hidden contextual information.
  arXiv  Detail & Related papers  (2026-02-05T06:24:23Z)
• Over-Searching in Search-Augmented Large Language Models [22.821710825732563]
  Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval.<n>Over-searching leads to computational inefficiency and hallucinations by incorporating irrelevant context.<n>Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention.
  arXiv  Detail & Related papers  (2026-01-09T03:24:46Z)
• SmartSearch: Process Reward-Guided Query Refinement for Search Agents [63.46067892354375]
  Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems.<n>Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked.<n>We introduce SmartSearch, a framework built upon two key mechanisms to mitigate this issue.
  arXiv  Detail & Related papers  (2026-01-08T12:39:05Z)
• Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Search [56.78490647843876]
  Agentic search has emerged as a promising paradigm for complex information seeking by enabling Large Language Models (LLMs) to interleave reasoning with tool use.<n>We propose bfM-ASK, a framework that explicitly decouples agentic search into two complementary roles: Search Behavior Agents, which plan and execute search actions, and Knowledge Management Agents, which aggregate, filter, and maintain a compact internal context.
  arXiv  Detail & Related papers  (2026-01-08T08:13:27Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• RE-Searcher: Robust Agentic Search with Goal-oriented Planning and Self-reflection [55.125987985864896]
  We present a systematic analysis that quantifies how environmental complexity induces fragile search behaviors.<n>We propose a simple yet effective approach to instantiate a search agent, RE-Searcher.<n>This combination of goal-oriented planning and self-reflection enables RE-Searcher to resist spurious cues in complex search environments.
  arXiv  Detail & Related papers  (2025-09-30T10:25:27Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• A Study on Leveraging Search and Self-Feedback for Agent Reasoning [16.256600534996686]
  We investigate how search and model's self-feedback can be leveraged for reasoning tasks.<n>First, we explore differences in ground-truth feedback and self-feedback during search for math reasoning.
  arXiv  Detail & Related papers  (2025-02-17T18:12:36Z)
• A Survey on Complex Tasks for Goal-Directed Interactive Agents [60.53915548970061]
  This survey compiles relevant tasks and environments for evaluating goal-directed interactive agents. An up-to-date compilation of relevant resources can be found on our project website.
  arXiv  Detail & Related papers  (2024-09-27T08:17:53Z)
• Beyond Semantics: Learning a Behavior Augmented Relevance Model with Self-supervised Learning [25.356999988217325]
  Relevance modeling aims to locate desirable items for corresponding queries. auxiliary query-item interactions extracted from user historical behavior data could provide hints to reveal users' search intents further. Our model builds multi-level co-attention for distilling coarse-grained and fine-grained semantic representations from both neighbor and target views.
  arXiv  Detail & Related papers  (2023-08-10T06:52:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.