DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows

• URL: http://arxiv.org/abs/2512.04416v2
• Date: Sat, 06 Dec 2025 06:46:21 GMT
• Title: DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
• Authors: Zhou Liu, Zhaoyang Han, Guochen Yan, Hao Liang, Bohan Zeng, Xing Chen, Yuanfeng Song, Wentao Zhang,
• Abstract summary: Large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into code.<n>Existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics.<n>We introduce DataGovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases.
• Score: 22.16698382751559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data governance ensures data quality, security, and compliance through policies and standards, a critical foundation for scaling modern AI development. Recently, large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into executable transformation code. However, existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics, failing to capture the unique challenge of data governance: ensuring the correctness and quality of the data itself. To bridge this gap, we introduce DataGovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases. DataGovBench employs a novel "reversed-objective" methodology to synthesize realistic noise and utilizes rigorous metrics to assess end-to-end pipeline reliability. Our analysis on DataGovBench reveals that current models struggle with complex, multi-step workflows and lack robust error-correction mechanisms. Consequently, we propose DataGovAgent, a framework utilizing a Planner-Executor-Evaluator architecture that integrates constraint-based planning, retrieval-augmented generation, and sandboxed feedback-driven debugging. Experimental results show that DataGovAgent significantly boosts the Average Task Score (ATS) on complex tasks from 39.7 to 54.9 and reduces debugging iterations by over 77.9 percent compared to general-purpose baselines.

Related papers

• Towards Compositional Generalization in LLMs for Smart Contract Security: A Case Study on Reentrancy Vulnerabilities [35.39583123277091]
  This paper proposes a post-training algorithm based on atomic task decomposition and fusion.<n>We decompose the reentrancy vulnerability detection task into four linearly independent atomic tasks.<n>By training on synthetic datasets, we generate three compiler-verified datasets.<n>We then employ the Slither tool to extract structural information from the control flow graph and data flow graph.
  arXiv  Detail & Related papers  (2026-01-11T13:52:07Z)
• Error-Driven Prompt Optimization for Arithmetic Reasoning [0.0]
  We introduce an error-driven optimization framework for arithmetic reasoning that enhances a Code Generation Agent (CGA)<n>We find that while the base model exhibits fundamental limitations in arithmetic tasks, our proposed error-driven method, which clusters erroneous predictions, dramatically improves performance.<n>Our results suggest that developing reliable, interpretable, and industrially deployable AI assistants can be achieved not only through costly fine-tuning but also via systematic, error-driven prompt optimization.
  arXiv  Detail & Related papers  (2025-12-15T13:39:14Z)
• Scaling Generalist Data-Analytic Agents [95.05161133349242]
  DataMind is a scalable data synthesis and agent training recipe designed to build generalist data-analytic agents.<n>DataMind tackles three key challenges in building open-source data-analytic agents.
  arXiv  Detail & Related papers  (2025-09-29T17:23:08Z)
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $τ$-bench [58.114899897566964]
  In a multi-turn conversational environment, large language models (LLMs) often struggle with consistent reasoning and adherence to domain-specific policies.<n>We propose the Input-Reformulation Multi-Agent (IRMA) framework, which automatically reformulates user queries augmented with relevant domain rules.<n>IRMA significantly outperforms ReAct, Function Calling, and Self-Reflection by 16.1%, 12.7%, and 19.1%, respectively.
  arXiv  Detail & Related papers  (2025-08-28T15:57:33Z)
• NatureGAIA: Pushing the Frontiers of GUI Agents with a Challenging Benchmark and High-Quality Trajectory Dataset [16.676904484703]
  We introduce NaturalGAIA, a novel benchmark engineered on the principle of Causal Pathways.<n>This paradigm structures complex tasks into a series of verifiable atomic steps, ensuring rigorous, fully automated, and reproducible standard for assessment.<n>We then utilize this dataset to perform Reinforcement FineTuning (RFT) on the Q2.5-VL-7B model.
  arXiv  Detail & Related papers  (2025-08-02T11:53:41Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• DABstep: Data Agent Benchmark for Multi-step Reasoning [2.6709582216950767]
  DABstep is a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks.<n>It comprises over 450 real-world challenges derived from a financial analytics platform.<n>Dabstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
  arXiv  Detail & Related papers  (2025-06-30T10:49:21Z)
• Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution [66.11004226578771]
  Existing robust benchmark datasets have two key limitations.<n>They generate only a limited range of perturbations for a single Information Extraction (IE) task.<n>Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench.<n>We show that training with only textbf15% of the data leads to an average textbf7.5% relative performance improvement across three IE tasks.
  arXiv  Detail & Related papers  (2025-03-05T05:39:29Z)
• RAAD-LLM: Adaptive Anomaly Detection Using LLMs and RAG Integration [2.879328762187361]
  We present RAAD-LLM, a novel framework for adaptive anomaly detection.<n>By effectively utilizing domain-specific knowledge, RAAD-LLM enhances the detection of anomalies in time series data.<n>Results show significant improvements over our previous model with an accuracy increase from 70.7% to 88.6% on the real-world dataset.
  arXiv  Detail & Related papers  (2025-03-04T17:20:43Z)
• DataSciBench: An LLM Agent Benchmark for Data Science [33.3811507234528]
  DataSciBench is a benchmark for evaluating Large Language Model (LLM) capabilities in data science.<n>We develop a semi-automated pipeline for generating ground truth (GT) and validating evaluation metrics.<n>We propose an innovative Task - Function - Code framework to assess each code execution outcome.
  arXiv  Detail & Related papers  (2025-02-19T17:31:51Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? [58.330879414174476]
  We introduce DSBench, a benchmark designed to evaluate data science agents with realistic tasks.<n>This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions.<n>Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG)
  arXiv  Detail & Related papers  (2024-09-12T02:08:00Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• DCA-Bench: A Benchmark for Dataset Curation Agents [9.60250892491588]
  Data quality issues, such as incomplete documentation, inaccurate labels, ethical concerns, and outdated information, remain common in widely used datasets.<n>With the surging ability of large language models (LLM), it's promising to streamline the discovery of hidden dataset issues with LLM agents.<n>In this work, we establish a benchmark to measure LLM agent's ability to tackle this challenge.
  arXiv  Detail & Related papers  (2024-06-11T14:02:23Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models [81.27391252152199]
  Large language models (LLMs) have achieved impressive performance across various natural language benchmarks. We propose to automate dataset updating and provide systematic analysis regarding its effectiveness. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, and 2) extending strategy that further expands existing samples.
  arXiv  Detail & Related papers  (2024-02-19T07:15:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.