Can Generalist Agents Automate Data Curation?

• URL: http://arxiv.org/abs/2606.04261v1
• Date: Tue, 02 Jun 2026 22:26:53 GMT
• Title: Can Generalist Agents Automate Data Curation?
• Abstract summary: Curating training data is among the most consequential yet labor-intensive parts of modern AI development.<n>We ask whether generalist coding agents can automate this data-curation loop.<n>We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite.
• Score: 58.652262227632406
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.

Related papers

• RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement [27.641105176473136]
  Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation.<n>We introduce RSIBench-Data, a benchmark of LLM agents as data-centric researchers with a fixed post-training stack.<n>We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics.
  arXiv  Detail & Related papers  (2026-07-28T15:46:41Z)
• Co-Evolving LLM Evaluators and Policies via DynamicRubric [67.2962847914162]
  Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.<n>We propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework.<n>In experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
  arXiv  Detail & Related papers  (2026-07-22T12:35:40Z)
• Unsupervised Skill Discovery for Agentic Data Analysis [55.42853694128929]
  DataCOPE is an unsupervised verifier-guided skill discovery framework for data-analytic agents.<n>It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation.<n>We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep.
  arXiv  Detail & Related papers  (2026-06-04T17:20:47Z)
• AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents [21.086121542846907]
  We present AgentDisCo, a novel Disentangled and Collaborative agentic architecture that formulates deep research as an adversarial optimization problem.<n>AgentDisCo employs a critic agent to evaluate generated outlines and refine search queries, and a generator agent to retrieve updated results and revise outlines accordingly.<n>The overall workflow supports both handcrafted and automatically discovered design strategies.
  arXiv  Detail & Related papers  (2026-05-12T08:14:15Z)
• AgentSim: A Platform for Verifiable Agent-Trace Simulation [3.2058241360543254]
  AgentSim is an open-source platform for simulating RAG agents.<n>It generates verifiable, stepwise traces of agent reasoning over any document collection.<n>It combines a multi-model validation pipeline with an active human-in-the-loop process.
  arXiv  Detail & Related papers  (2026-04-29T13:19:38Z)
• Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs [28.877758891217184]
  RADAR is a framework for comparing test-time evaluation tasks to policy training data.<n>We show that vision-language models (VLMs) are effective at analyzing data for generalization.<n>We scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions.
  arXiv  Detail & Related papers  (2026-03-12T01:33:06Z)
• What's the next frontier for Data-centric AI? Data Savvy Agents [71.76058707995398]
  We argue that data-savvy capabilities should be a top priority in the design of agentic systems.<n>We propose four key capabilities to realize this vision: Proactive data acquisition, Sophisticated data processing, Interactive test data synthesis, and Continual adaptation.
  arXiv  Detail & Related papers  (2025-11-02T17:09:29Z)
• Issue-Oriented Agent-Based Framework for Automated Review Comment Generation [15.04868140672973]
  RevAgent is a novel agent-based issue-oriented framework for code review comments.<n>It decomposes the task into three stages: Generation, Discrimination, and Training.<n>It significantly outperforms state-of-the-art PLM- and LLM-based baselines.
  arXiv  Detail & Related papers  (2025-11-01T11:44:11Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• Foundation Policies with Hilbert Representations [54.44869979017766]
  We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data. Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment. Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2024-02-23T19:09:10Z)
• Explaining Reinforcement Learning Policies through Counterfactual Trajectories [147.7246109100945]
  A human developer must validate that an RL agent will perform well at test-time. Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
  arXiv  Detail & Related papers  (2022-01-29T00:52:37Z)
• Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline [94.0601799665342]
  Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summarization task. We propose establishing summary-source alignment as an explicit task, while introducing two major novelties. We create a novel training dataset for proposition-level alignment, derived automatically from available summarization evaluation data. We present a supervised proposition alignment baseline model, showing improved alignment-quality over the unsupervised approach.
  arXiv  Detail & Related papers  (2020-09-01T17:27:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.