Benchmarking Hybrid Deep Research Across Database Querying and Web Search

• URL: http://arxiv.org/abs/2609.09410v1
• Date: Tue, 08 Sep 2026 20:05:09 GMT
• Title: Benchmarking Hybrid Deep Research Across Database Querying and Web Search
• Abstract summary: HybridDeepResearch is first benchmark that requires both web search and relational databases to form a complete, verifiable answer.<n>State-of-the-art models like GLM-5.2, Claude-Sonnet-4.6 and GPT-5 achieve only about 50-54% Pass@8 on the hard subset.<n>Results show that directional reasoning is substantially more difficult than parallel intersection.
• Score: 35.12550152156569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While autonomous agents have made significant strides in "deep research" by iteratively navigating the open web to synthesize information, real-world problem-solving is rarely confined to a single environment. Complex analytical tasks inherently require agents to weave together evidence from both ambiguous unstructured text (e.g., the open web) and highly precise structured data (e.g., relational databases). However, existing benchmarks evaluate these modalities in isolation, failing to capture the critical "handoff" - the ability to preserve constraints when moving evidence between systems. We introduce HybridDeepResearch, to our knowledge the first deep-research benchmark that requires both web search and SQL to form a complete, verifiable answer. The benchmark contains 380 tool-dependent tasks grounded in LiveSQLBench-Base-Lite databases and public web corpora, validated through automated checks and human review, and covering three reasoning patterns: SQL2S, S2SQL, and Parallel. Evaluations across proprietary and open-weight models under various agentic scaffolds reveal that even state-of-the-art models like GLM-5.2, Claude-Sonnet-4.6 and GPT-5 achieve only about 50-54% Pass@8 on the hard subset. Notably, results show that directional reasoning is substantially more difficult than parallel intersection, highlighting that bridging structured and unstructured information spaces without losing constraints remains a major open challenge for agentic systems. Code and datasets are publicly available at GitHub (https://github.com/Snowflake-AI-Research/HybridDeepResearch) and Hugging Face (https://huggingface.co/datasets/Snowflake/HybridDeepResearch).

Related papers

• DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning [58.09293442211248]
  Deep Research (DR) has emerged as a new agentic paradigm to tackle complex, open-ended research tasks.<n>This report presents DuMate-DeepResearch, a multi-agent DR framework built on the Qianfan Agent Foundry.
  arXiv  Detail & Related papers  (2026-06-05T14:10:48Z)
• Web2BigTable: A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction [38.55682016152131]
  textbfWeb2BigTable is a multi-agent framework for web-to-table search.<n>Web2BigTable sets a new state of the art on WideSearch, reaching an Avg@4 Success Rate of textbf38.50 ($7.5times$ the second best at 5.10), Row F1 of textbf63.53 (+25.03 over the second best) and Item F1 of textbf80.12 (+14.42 over the second best)<n>It also generalises to depth-oriented search on XBench
  arXiv  Detail & Related papers  (2026-04-29T21:43:16Z)
• ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget [56.6320292944541]
  ORBIT is a training dataset with 20K reasoning-intensive queries with short verifiable answers.<n>Our framework, code and datasets are open-sourced and available publicly.
  arXiv  Detail & Related papers  (2026-04-01T17:42:41Z)
• APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQL [39.76924093980244]
  APEX- verbalize is a framework that shifts the paradigm from passive translation to agentic exploration.<n>Our framework employs a hypothesis-verification loop to ground model reasoning in real data.
  arXiv  Detail & Related papers  (2026-02-11T07:50:47Z)
• Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analytics [75.4712507893024]
  Enterprise Deep Research (EDR) is a multi-agent system that integrates a Master Planning Agent for adaptive query decomposition.<n>Four specialized search agents (General, Academic, GitHub, LinkedIn) and a visualization agent for data-driven insights are also included.<n>EDR reflects research direction with optional human-in-the-loop steering guidance.
  arXiv  Detail & Related papers  (2025-10-20T17:55:11Z)
• Open Data Synthesis For Deep Research [17.22470203913576]
  We formalize Deep Research tasks with verifiable answers as Hierarchical Constraint Satisfaction Problems.<n>Existing benchmarks (e.g., Natural Questions, HotpotQA) fail to capture this complexity.<n>We introduce InfoSeek, a scalable scalable complex Deep Research tasks.
  arXiv  Detail & Related papers  (2025-08-30T06:02:56Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• Text2VectorSQL: Towards a Unified Interface for Vector Search and SQL Queries [36.92547259037192]
  The proliferation of unstructured data poses a fundamental challenge to traditional database infrastructure.<n>While Text-to-BIRD has democratized access to structured data, it remains incapable of interpreting semantic or multi-modal queries.<n>We introduce and formalize Text2 Vector, a novel task to establish a unified natural language for seamlessly querying both structured and unstructured data.
  arXiv  Detail & Related papers  (2025-06-29T03:17:42Z)
• DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning [73.68685269970844]
  We introduce WebPuzzle, a training and 275-sample test benchmark that evaluates information seeking on the live internet.<n>We develop DeepDiver, a reinforcement-learning framework that cultivates Search Intensity Scaling (SIS)-an emergent ability to escalate search frequency and depth.<n>We detail DeepDiver's curriculum from cold-start SFT to a well designed RL procedure, and show that its seeking policy generalized from closed-ended queries to open-ended generation such as long-form writing.
  arXiv  Detail & Related papers  (2025-05-30T08:15:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.