SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval

• URL: http://arxiv.org/abs/2605.22219v1
• Date: Thu, 21 May 2026 09:22:48 GMT
• Title: SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
• Abstract summary: A class of specialized retrieval tasks remains undercharacterized.<n>We term this capability state-gated retrieval (SGR)<n>SGR-Bench is a benchmark for this setting containing 100 expert-curated tasks spanning six source families and 12 public data ecosystems.
• Score: 5.32888938421653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in large language models and tool-using agents have expanded the range of benchmarked web tasks. Yet an important class of specialized retrieval tasks remains undercharacterized. On many specialized data-retrieval websites, answer-bearing evidence becomes accessible only after establishing the correct site-specific retrieval state through filters, views, hierarchies, or scopes. We term this capability state-gated retrieval (SGR). We introduce SGR-Bench, a benchmark for this setting containing 100 expert-curated tasks spanning six source families and 12 public data ecosystems. Each task requires discovering the appropriate website and configuring its site-specific retrieval state to produce a structured answer. SGR-Bench pairs constraint-guided and goal-oriented formulations of the same underlying problems, enabling controlled comparisons between explicit and implicit guidance for state-gated retrieval. We evaluate eight CLI-based agentic LLM systems and three commercial search-agent products. On SGR-Bench, the strongest system reaches only 66.18% item-level F1, while row-level F1 remains much lower. A manual audit of 156 analyzable failed CLI trajectories shows why: agents often reach a relevant web source, but establish the wrong site-specific retrieval state. Retrieval-scope drift (37.2%) and criterion mismatch (27.6%) dominate, whereas final answer composition accounts for only 10.3%. The dataset and single-case evaluation instructions are available at https://huggingface.co/datasets/PKUAIWeb/SGR-BENCH.

Related papers

• CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval [20.44841712209369]
  CaSKG is a skill graph framework that calibrates procedural relations before retrieval.<n>It achieves the highest task score in all twelve combinations of model and benchmark.
  arXiv  Detail & Related papers  (2026-08-26T08:12:41Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• FinSAgent: Corpus-Aligned Multi-Agent RAG Framework for Evidence-Grounded SEC Filing Question Answering [16.415865984091667]
  FinSAgent is an evidence-grounded multi-agent framework that reframes SEC filing QA as corpus-aligned retrieval planning.<n>FinSAgent improves retrieval coverage and answer correctness over strong single-agent and multi-agent baselines.
  arXiv  Detail & Related papers  (2026-07-20T16:03:15Z)
• Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval [3.6202634353482352]
  The rise of Large Language Models (LLMs) capable of navigating the unstructured web raises a fundamental question: Is semantic metadata still necessary for agentic data discovery?<n>We present a comparative analysis of agentic data retrieval across two distinct environments: a Baseline Agent searching billions of open-web documents, and a Semantic Agent leveraging a corpus of 90 million datasets using schema.org.<n>The Semantic Agent excels at retrieving actionable data, achieving a 44.9% higher precision for metadata-rich registries and a 46.6% higher precision for pages with machine-readable downloads among its returned results.
  arXiv  Detail & Related papers  (2026-05-27T17:46:43Z)
• EfficientGraph-RAG: Structured Retrieval-State Management for Cross-Task Retrieval-Augmented Generation [45.66803833786235]
  Retrieval-augmented generation has become the standard way to ground large language models in external knowledge.<n> EfficientGraph-RAG ranks first on the reported answer-quality metrics averaged over the three evaluated LongBench retrieval-style subsets.
  arXiv  Detail & Related papers  (2026-05-25T03:08:53Z)
• Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
  Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
  arXiv  Detail & Related papers  (2026-03-09T18:20:20Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views [24.456069497637035]
  Vector Similarity Search (VSS) in high-dimensional spaces is rapidly emerging as core functionality in next-generation database systems.<n>Current benchmarks evaluate VSS primarily on the recall-latency trade-off against a ground truth defined solely by distance metrics.<n>We present Iceberg, a holistic benchmark suite for end-to-end evaluation of VSS methods in realistic application contexts.
  arXiv  Detail & Related papers  (2025-12-15T04:49:33Z)
• Clue-RAG: Towards Accurate and Cost-Efficient Graph-based RAG via Multi-Partite Graph and Query-Driven Iterative Retrieval [15.599544326509436]
  Retrieval-Augmented Generation (RAG) addresses the limitation by incorporating external information, often from graph-structured data.<n>We propose Clue-RAG, a novel approach that introduces a multi-partite graph index and a query-driven iterative retrieval strategy.<n>Experiments on three QA benchmarks show that Clue-RAG significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-07-11T09:36:45Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• Tree-Based Text Retrieval via Hierarchical Clustering in RAGFrameworks: Application on Taiwanese Regulations [0.0]
  We propose a hierarchical clustering-based retrieval method that eliminates the need to predefine k.<n>Our approach maintains the accuracy and relevance of system responses while adaptively selecting semantically relevant content.<n>Our framework is simple to implement and easily integrates with existing RAG pipelines, making it a practical solution for real-world applications under limited resources.
  arXiv  Detail & Related papers  (2025-06-16T15:34:29Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.