AIDABench: AI Data Analytics Benchmark

• URL: http://arxiv.org/abs/2603.15636v1
• Date: Fri, 27 Feb 2026 08:58:05 GMT
• Title: AIDABench: AI Data Analytics Benchmark
• Abstract summary: AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
• Score: 62.45908988324612
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI-driven document understanding and processing tools become increasingly prevalent in real-world applications, the need for rigorous evaluation standards has grown increasingly urgent. Existing benchmarks and evaluations often focus on isolated capabilities or simplified scenarios, failing to capture the end-to-end task effectiveness required in practical settings. To address this gap, we introduce AIDABench, a comprehensive benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner. AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation. These tasks are grounded in realistic scenarios involving heterogeneous data types, including spreadsheets, databases, financial reports, and operational records, and reflect analytical demands across diverse industries and job functions. Notably, the tasks in AIDABench are sufficiently challenging that even human experts require 1-2 hours per question when assisted by AI tools, underscoring the benchmark's difficulty and real-world complexity. We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families. Our results reveal that complex, real-world data analytics tasks remain a significant challenge for current AI systems, with the best-performing model achieving only 59.43% pass-at-1. We provide a detailed analysis of failure modes across each capability dimension and identify key challenges for future research. AIDABench offers a principled reference for enterprise procurement, tool selection, and model optimization, and is publicly available at https://github.com/MichaelYang-lyx/AIDABench.

Related papers

• AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs [55.8213573429699]
  AISE-Bench is a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs.<n>AISE-Bench release contains 1,133 QA pairs, including query, full API execution, validated parameters, and source-grounded answers with reference links.<n>We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success.
  arXiv  Detail & Related papers  (2026-06-16T07:59:46Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle [41.576417987200074]
  Real-world enterprise data intelligence encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights.<n>We introduce DAComp, a benchmark of 210 tasks that mirrors these complex capabilities.
  arXiv  Detail & Related papers  (2025-12-03T23:21:28Z)
• OIDA-QA: A Multimodal Benchmark for Analyzing the Opioid Industry Documents Archive [50.468138755368805]
  Opioid crisis represents a significant moment in public health.<n>Data and documents disclosed in the UCSF-JHU Opioid Industry Documents Archive (OIDA)<n>In this paper, we tackle this challenge by organizing the original dataset according to document attributes.
  arXiv  Detail & Related papers  (2025-11-13T03:27:32Z)
• Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures [0.0]
  We replicate an existing, challenging real-world annotation task with complex multi-part criteria on thousands of pages of public disclosure documents.<n>We find that when extrapolated to the full task, these methods could save up to 268 hours compared to the human-only approach.
  arXiv  Detail & Related papers  (2025-07-28T22:06:11Z)
• DABstep: Data Agent Benchmark for Multi-step Reasoning [2.6709582216950767]
  DABstep is a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks.<n>It comprises over 450 real-world challenges derived from a financial analytics platform.<n>Dabstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
  arXiv  Detail & Related papers  (2025-06-30T10:49:21Z)
• TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents [17.296425855109426]
  We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents.<n>TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and tasks.<n>We implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models.
  arXiv  Detail & Related papers  (2025-05-19T16:11:23Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)
• InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation [79.09622602860703]
  We introduce InsightBench, a benchmark dataset with three key features.<n>It consists of 100 datasets representing diverse business use cases such as finance and incident management.<n>Unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics.
  arXiv  Detail & Related papers  (2024-07-08T22:06:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.