AgenticDataBench: A Comprehensive Benchmark for Data Agents

• URL: http://arxiv.org/abs/2607.01647v1
• Date: Thu, 02 Jul 2026 03:18:59 GMT
• Title: AgenticDataBench: A Comprehensive Benchmark for Data Agents
• Abstract summary: Large language model (MLL) data agents have emerged as a promising solution to automate data science.<n>We propose AgenticData, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained granularity.<n>This enables evaluations to capture the diversity and complexity of data science and the detailed performance of agents.
• Score: 26.292243502468335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.

Related papers

• DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? [40.322111941653134]
  We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science inside real computer environments.<n>Our experiments show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success.<n>These results reveal a substantial capability gap between current agentic systems and real data-science.
  arXiv  Detail & Related papers  (2026-08-11T01:45:56Z)
• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs [66.63911043019294]
  Data preparation aims to denoise raw datasets, uncover cross-dataset relationships, and extract valuable insights from them.<n>This paper focuses on the use of LLM techniques to prepare data for diverse downstream tasks.<n>We introduce a task-centric taxonomy that organizes the field into three major tasks: data cleaning, standardization, error processing, imputation, data integration, and data enrichment.
  arXiv  Detail & Related papers  (2026-01-22T12:02:45Z)
• CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories [15.512057716487517]
  We propose CRMWeaver, a novel approach that enhances business agents in complex settings.<n>We employ a synthesis data generation and RL-based paradigm during training, which significantly improves the model's ability to handle complex data.<n>We validate the efficacy of our approach on the CRMArena-Pro dataset, underscoring its practical value for real-world applications.
  arXiv  Detail & Related papers  (2025-10-29T09:47:40Z)
• Scaling Generalist Data-Analytic Agents [95.05161133349242]
  DataMind is a scalable data synthesis and agent training recipe designed to build generalist data-analytic agents.<n>DataMind tackles three key challenges in building open-source data-analytic agents.
  arXiv  Detail & Related papers  (2025-09-29T17:23:08Z)
• DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation [10.390461679868197]
  We introduce DatawiseAgent, a notebook-centric large language model (LLM) agent framework for adaptive and robust data science automation.<n>Inspired by how human data scientists work in computational notebooks, DatawiseAgent introduces a unified interaction representation and a multi-stage architecture.
  arXiv  Detail & Related papers  (2025-03-10T08:32:33Z)
• Making Sense of Data in the Wild: Data Analysis Automation at Scale [0.1747623282473278]
  We propose a novel approach that combines intelligent agents with retrieval augmented generation to automate data analysis, dataset curation and indexing at scale.<n>We demonstrate that our approach results in more detailed dataset descriptions, higher hit rates and greater diversity in dataset retrieval tasks.
  arXiv  Detail & Related papers  (2025-01-27T10:04:10Z)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? [58.330879414174476]
  We introduce DSBench, a benchmark designed to evaluate data science agents with realistic tasks.<n>This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions.<n>Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG)
  arXiv  Detail & Related papers  (2024-09-12T02:08:00Z)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems [10.71630696651595]
  Compound AI systems (CASs) that employ LLMs as agents to accomplish knowledge-intensive tasks have garnered significant interest within database and AI communities. silos of multimodal data sources make it difficult to identify appropriate data sources for accomplishing the task at hand. We propose CMDBench, a benchmark modeling the complexity of enterprise data platforms.
  arXiv  Detail & Related papers  (2024-06-02T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.