From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

• URL: http://arxiv.org/abs/2608.02163v1
• Date: Mon, 03 Aug 2026 12:43:00 GMT
• Title: From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
• Abstract summary: We introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories.<n>The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline.<n>Our data, implementation, and results are publicly available.
• Score: 14.763650534859176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.

Related papers

• Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents [16.43230671992556]
  AutoSciRub is an evaluation-first framework that induces a task-specific executable before research execution.<n>AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria.<n>During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report.
  arXiv  Detail & Related papers  (2026-08-31T16:48:51Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design [39.31356016375221]
  Marco DeepResearch is a deep research agent optimized with a verification-centric framework design at three levels.<n>We introduce verification mechanisms to graph-based and agent-based QA synthesis to control question difficulty.<n>We design a verification-driven synthesis trajectory method that injects explicit verification patterns into training trajectories.<n>We use Marco DeepResearch itself as a verifier at inference time and effectively improve performance on challenging questions.
  arXiv  Detail & Related papers  (2026-03-30T12:42:02Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents [10.197402632091551]
  DeepSearchQA is a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks.<n>This dataset is designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists.
  arXiv  Detail & Related papers  (2026-01-28T19:20:47Z)
• SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback [68.60326181052658]
  We propose an agentic pipeline that automatically generates high quality, difficulty-controlled deep search question-answer pairs.<n>Our pipeline, SAGE, consists of a data generator which proposes QA pairs and a search agent which attempts to solve the generated question.<n>Our intrinsic evaluation shows SAGE generates questions that require diverse reasoning strategies, while significantly increases the correctness and difficulty of the generated data.
  arXiv  Detail & Related papers  (2026-01-26T06:37:56Z)
• DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation [56.886936435727854]
  DeepResearchEval is an automated framework for deep research task construction and agentic evaluation.<n>For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles.<n>For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
  arXiv  Detail & Related papers  (2026-01-14T18:38:31Z)
• Towards Personalized Deep Research: Benchmarks and Evaluations [56.581105664044436]
  We introduce Personalized Deep Research Bench, the first benchmark for evaluating personalization in Deep Research Agents (DRAs)<n>It pairs 50 diverse research tasks with 25 authentic user profiles that combine structured persona attributes with dynamic real-world contexts, yielding 250 realistic user-task queries.<n>Our experiments on a range of systems highlight current capabilities and limitations in handling personalized deep research.
  arXiv  Detail & Related papers  (2025-09-29T17:39:17Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.