DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis

• URL: http://arxiv.org/abs/2605.02503v1
• Date: Mon, 04 May 2026 11:57:09 GMT
• Title: DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis
• Abstract summary: DataClaw is a process-oriented benchmark for exploratory real-world data analysis.<n>It contains approximately 2.06 million real-world records across enterprise, industry and policy domains.<n>DataClaw measures how far an agent progresses and where its reasoning breaks down.
• Score: 76.98578575566184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating autonomous data analysis agents requires testing their ability to perform exploratory analysis in underexplored data environments. However, many existing benchmarks emphasize final answer accuracy in prior-guided data settings and provide limited support for reasoning process evaluation. We introduce DataClaw, a process-oriented benchmark for exploratory real-world data analysis. DataClaw contains approximately 2.06 million real-world records across enterprise, industry and policy domains, with native data noise preserved. It further includes 492 cross-domain tasks derived from think-tank consulting scenarios, each annotated with intermediate milestones for process-level evaluation. These annotations allow DataClaw to measure how far an agent progresses and where its reasoning breaks down. Experiments with eight advanced LLMs show that current agents remain far from reliable in this setting, with seven models achieving below 50% overall accuracy. Process analysis further reveals partial progress hidden behind wrong answers and distinct exploration strategies across models. Overall, DataClaw provides a less data constrained diagnostic testbed for probing the capability boundaries of autonomous data-analysis agents.

Related papers

• Unsupervised Skill Discovery for Agentic Data Analysis [55.42853694128929]
  DataCOPE is an unsupervised verifier-guided skill discovery framework for data-analytic agents.<n>It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation.<n>We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep.
  arXiv  Detail & Related papers  (2026-06-04T17:20:47Z)
• AgentSCOPE: Evaluating Contextual Privacy Across Agentic Workflows [32.40442323501255]
  We argue that every boundary in an agentic pipeline is a site of potential privacy violation and must be assessed independently.<n>We introduce the Privacy Flow Graph, a framework that decomposes agentic execution into a sequence of information flows.<n>We present AgentSCOPE, a benchmark of 62 multi-tool scenarios across eight regulatory domains with ground truth at every pipeline stage.
  arXiv  Detail & Related papers  (2026-03-05T07:45:31Z)
• A Benchmark for Deep Information Synthesis [27.87976801124957]
  DEEPSYNTH is a novel benchmark designed to evaluate large language model (LLM)-based agents on realistic, time-consuming problems.<n>It contains 120 tasks collected across 7 domains and data sources covering 67 countries.<n>When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric.
  arXiv  Detail & Related papers  (2026-02-24T17:43:32Z)
• LongDA: Benchmarking LLM Agents for Long-Document Data Analysis [55.32211515932351]
  LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck.<n>LongTA is a tool-augmented agent framework that enables document access, retrieval, and code execution.<n>Our experiments reveal substantial performance gaps even among state-of-the-art models.
  arXiv  Detail & Related papers  (2026-01-05T23:23:16Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• DeepAnalyze: Agentic Large Language Models for Autonomous Data Science [35.69385623867138]
  We introduce DeepAnalyze-8B, the first agentic designed for autonomous data science.<n>We propose a curriculum-based agentic training paradigm that emulates the learning trajectory of human data scientists.<n>We also introduce a data-grounded trajectory synthesis framework that constructs high-quality training data.
  arXiv  Detail & Related papers  (2025-10-19T15:13:42Z)
• DABstep: Data Agent Benchmark for Multi-step Reasoning [2.6709582216950767]
  DABstep is a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks.<n>It comprises over 450 real-world challenges derived from a financial analytics platform.<n>Dabstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
  arXiv  Detail & Related papers  (2025-06-30T10:49:21Z)
• DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation [83.30006900263744]
  Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases.
  arXiv  Detail & Related papers  (2024-03-04T22:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.