Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search

• URL: http://arxiv.org/abs/2608.20317v1
• Date: Thu, 20 Aug 2026 17:54:48 GMT
• Title: Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search
• Abstract summary: The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus.<n>We introduce $textBrowseComp-Plus_textCM$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix.<n>Our main contribution is the projection pipeline that makes this possible.
• Score: 40.897106650424035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus, so that an agent's role can be separated from the retriever's. That corpus, however, holds only about 100K documents and was assembled from the supporting documents of the benchmark's own queries plus mined hard negatives, so the evidence and the distractors were both selected per query. We introduce $\text{BrowseComp-Plus}_{\text{CM}}$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix, a 400B-token, 553M-document mixture of web text released by NVIDIA for pre-training language models and built without reference to any benchmark. Our main contribution is the projection pipeline that makes this possible: it decomposes each question into atomic reasoning hops and grounds every hop in the new corpus, retaining a question only when automatic verification, an independent agent, and human review all confirm that every hop is supported. The pipeline is dataset-agnostic and applies to any benchmark whose questions decompose into verifiable facts. Applied to the 830 BrowseComp-Plus test questions, our pipeline yields 57 fully grounded questions with question-level relevance judgments. Projection shifts the difficulty onto retrieval, as the strongest agent we evaluate loses five points of answer accuracy but sees its evidence recall fall from 84.3% to 21.4% while issuing 63% more search calls. As the first of a series of projections, we release the pipeline, the benchmark, and our analyses at https://github.com/castorini/cmass.

Related papers

• SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs [30.494397052362384]
  We introduce SearchAtlas, a framework that converts search trajectories into structured graphs.<n>Our automated parsing pipeline achieves a mean edge F1 of 86.4% against human-annotated graphs.<n>We analyze five search agents on three benchmarks, revealing systematic differences in search scale and evidence aggregation.
  arXiv  Detail & Related papers  (2026-09-09T23:09:26Z)
• Iris: Climbing to the Search Frontier [21.066541702697112]
  We present Iris-mini and Iris-pro, two search agents trained at the 35B-A3B and 397B-A17B scales, together with the data pipeline and training recipe behind them.
  arXiv  Detail & Related papers  (2026-09-03T17:51:01Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning [52.23991730630292]
  Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge.<n>We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer.<n>We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window.
  arXiv  Detail & Related papers  (2026-06-23T12:57:18Z)
• EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent [8.1223119066448]
  EComAgentBench is a benchmark of 662 tasks grounded in real Amazon products and reviews.<n>Each task scatters requirements across a visible query, a tool-gated profile, and scripted clarification.<n> typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source.
  arXiv  Detail & Related papers  (2026-06-16T09:10:48Z)
• LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know? [32.434901767447165]
  LLM-based search agents rely on intrinsic knowledge rather than external evidence.<n>LiveBrowseComp is a benchmark designed to evaluate agents beyond intrinsic coverage.<n>All evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance.
  arXiv  Detail & Related papers  (2026-05-27T16:39:57Z)
• ProactBench: Beyond What The User Asked For [5.422521416406412]
  ProactBench decomposes it into three phase-tied types: textscEmergent, inference from a single disclosed anchor; textscCritical, synthesis across multiple anchors; and textscRecovery, grounded forward-looking value after task completion.<n>Our information asymmetries defend against style-confounding scoring, leakage, external-context contamination, and information dumps.
  arXiv  Detail & Related papers  (2026-05-09T23:56:04Z)
• SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation [0.0]
  We introduce SPD-RAG, a hierarchical multi-agent framework for exhaustive cross-document question answering.<n>Each document is processed by a dedicated document-level agent operating only on its own content, enabling focused retrieval.<n>SPD-RAG achieves an Avg Score of 58.1 (GPT-5 evaluation), outperforming Normal RAG (33.0) and Agentic RAG (32.8) while using only 38% of the API cost of a full-context baseline (68.0)
  arXiv  Detail & Related papers  (2026-03-09T12:46:32Z)
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent [74.10138164281618]
  BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.<n>This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.
  arXiv  Detail & Related papers  (2025-08-08T17:55:11Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.