DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

• URL: http://arxiv.org/abs/2608.03451v1
• Date: Tue, 04 Aug 2026 10:48:11 GMT
• Title: DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces
• Abstract summary: Data agents enable natural-language analytics over organizational workspaces.<n>Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis.<n>We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results.
• Score: 25.28248447900504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

Related papers

• Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement [0.0]
  We present a framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents.<n>A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents.<n>We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents.
  arXiv  Detail & Related papers  (2026-09-10T18:34:08Z)
• What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents [81.23160457293714]
  LLM agents increasingly rely on generated interaction data to learn how to interact with external environments.<n>Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms.<n>This work develops a two-level framework for the field.
  arXiv  Detail & Related papers  (2026-08-27T15:43:50Z)
• Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoning [0.41942958779358674]
  Existing evaluations focus on isolated single-platform environments.<n>We introduce the Cross-Vendor Sola ISPM Benchmark, a production-grade benchmark of 50 data-grounded tasks.<n>We also contribute an evaluation framework measuring not only final answer correctness but also evidentiary grounding.
  arXiv  Detail & Related papers  (2026-06-01T12:41:00Z)
• Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval [3.6202634353482352]
  The rise of Large Language Models (LLMs) capable of navigating the unstructured web raises a fundamental question: Is semantic metadata still necessary for agentic data discovery?<n>We present a comparative analysis of agentic data retrieval across two distinct environments: a Baseline Agent searching billions of open-web documents, and a Semantic Agent leveraging a corpus of 90 million datasets using schema.org.<n>The Semantic Agent excels at retrieving actionable data, achieving a 44.9% higher precision for metadata-rich registries and a 46.6% higher precision for pages with machine-readable downloads among its returned results.
  arXiv  Detail & Related papers  (2026-05-27T17:46:43Z)
• DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis [76.98578575566184]
  DataClaw is a process-oriented benchmark for exploratory real-world data analysis.<n>It contains approximately 2.06 million real-world records across enterprise, industry and policy domains.<n>DataClaw measures how far an agent progresses and where its reasoning breaks down.
  arXiv  Detail & Related papers  (2026-05-04T11:57:09Z)
• TADI: Tool-Augmented Drilling Intelligence via Agentic LLM Orchestration over Heterogeneous Wellsite Data [0.0]
  TADI (Tool-Augmented Drilling Intelligence) is an agentic AI system that transforms drilling operational data into evidence-based analytical intelligence.<n>TADI integrates 1,759 daily drilling reports, selected WIT real-time objects, 15,634 production records, formation tops, and perforations into a dual-store architecture.
  arXiv  Detail & Related papers  (2026-04-30T03:19:39Z)
• DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysis [8.171937411588015]
  We introduce DataCross, a novel benchmark and collaborative agent framework for unified, insight-driven analysis.<n>DataCrossBench comprises 200 end-to-end analysis tasks across finance, healthcare, and other domains.<n>We also propose the DataCrossAgent framework, inspired by the "divide-and-synthesis" workflow of human analysts.
  arXiv  Detail & Related papers  (2026-01-29T08:40:45Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• From Questions to Queries: An AI-powered Multi-Agent Framework for Spatial Text-to-SQL [0.4499833362998488]
  Single-agent approaches often struggle with the semantic and syntactic complexities of spatial queries.<n>We propose a multi-agent framework designed to accurately translate natural language questions into spatial queries.<n>We evaluate our system using both the non-spatial KaggleDBQA benchmark and a new, comprehensive SpatialQA benchmark.
  arXiv  Detail & Related papers  (2025-10-23T22:58:17Z)
• Scaling Generalist Data-Analytic Agents [95.05161133349242]
  DataMind is a scalable data synthesis and agent training recipe designed to build generalist data-analytic agents.<n>DataMind tackles three key challenges in building open-source data-analytic agents.
  arXiv  Detail & Related papers  (2025-09-29T17:23:08Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning [98.26836657967162]
  textbfAgentOhana aggregates agent trajectories from distinct environments, spanning a wide array of scenarios. textbfxLAM-v0.1, a large action model tailored for AI agents, demonstrates exceptional performance across various benchmarks.
  arXiv  Detail & Related papers  (2024-02-23T18:56:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.