PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?

• URL: http://arxiv.org/abs/2605.08687v1
• Date: Sat, 09 May 2026 04:52:47 GMT
• Title: PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?
• Abstract summary: Recent advances in large language models (LLMs) raise the possibility of a paradigm shift toward natural language (NL)-driven data preparation.<n>Existing code generation benchmarks do not capture key characteristics of data preparation.<n>We present PrepBench, a benchmark designed to evaluate NL-driven data preparation along three core capabilities.
• Score: 11.21730510420337
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data preparation is a central and time-consuming stage in data analysis workflows. Traditionally, commercial tools have relied on graphical user interfaces (GUIs) to simplify data preparation, allowing users to define transformations through visual operators and workflows. Recent advances in large language models (LLMs) raise the possibility of a paradigm shift toward natural language (NL)-driven data preparation, in which users can specify preparation intents in NL directly. However, it remains unclear how far current LLM-based agents are from this paradigm shift in practice. Existing code generation benchmarks do not capture key characteristics of data preparation, including ambiguous user intents, imperfect real-world data, and the need to translate code into interpretable workflows for validation. To bridge this gap, we present PrepBench, a benchmark designed to evaluate NL-driven data preparation along three core capabilities: interactive disambiguation, prep-code generation, and code-to-workflow translation. We crawl data from the Preppin' Data Challenges, and then extend it into a systematically designed benchmark. The benchmark covers diverse domains, and each task involves 3 to 18 data preparation steps. Nearly half of the tasks require over 100 lines of Python code, and the longest solutions approach 300 lines. Our evaluation shows that, despite recent progress, realizing this paradigm shift remains challenging for state-of-the-art LLMs. PrepBench provides a principled benchmark for measuring this gap and helps identify key challenges toward realizing NL-driven data preparation.

Related papers

• Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks [0.0]
  Large language models (LLMs) and agents are now widely used tools in code development.<n>Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services.<n>We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies.
  arXiv  Detail & Related papers  (2026-07-23T16:23:42Z)
• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs [66.63911043019294]
  Data preparation aims to denoise raw datasets, uncover cross-dataset relationships, and extract valuable insights from them.<n>This paper focuses on the use of LLM techniques to prepare data for diverse downstream tasks.<n>We introduce a task-centric taxonomy that organizes the field into three major tasks: data cleaning, standardization, error processing, imputation, data integration, and data enrichment.
  arXiv  Detail & Related papers  (2026-01-22T12:02:45Z)
• DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI [42.191938707504406]
  DataFlow is a unified and LLM-driven data preparation framework.<n>System-level abstractions enable modular, reusable, and composable data transformations.<n>DataFlow consistently improves downstream Large Language Models performance.
  arXiv  Detail & Related papers  (2025-12-18T15:46:15Z)
• LLM/Agent-as-Data-Analyst: A Survey [54.08761322298559]
  Large language models (LLMs) and agent techniques have brought a fundamental shift in the functionality and development paradigm of data analysis tasks.<n>LLMs enable complex data understanding, natural language, semantic analysis functions, and autonomous pipeline orchestration.
  arXiv  Detail & Related papers  (2025-09-28T17:31:38Z)
• LLM-Aided Customizable Profiling of Code Data Based On Programming Language Concepts [1.480144196276276]
  This work addresses profiling specifically in the context of code datasets for Large Language Models (code-LLMs)<n>Characterizing code datasets in terms of programming language concepts enables better insights and targeted data curation.
  arXiv  Detail & Related papers  (2025-03-19T11:01:00Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [91.52444946362547]
  We introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused Large Language Models and observed consistent improvements in performance on two widely-used programming skill benchmarks.
  arXiv  Detail & Related papers  (2024-02-20T13:56:38Z)
• From Words to Code: Harnessing Data for Program Synthesis from Natural Language [12.665932954069476]
  We introduce semantic reranking, a technique to rerank the programs generated by large language models (LLMs) We also introduce temperature mixing, where we combine samples generated by LLMs using both high and low temperatures. We observe substantial gains across domains, with improvements of up to 45% in top-1 accuracy and 34% in top-3 accuracy.
  arXiv  Detail & Related papers  (2023-05-02T16:56:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.