Automated Data Readiness for Scientific AI

• URL: http://arxiv.org/abs/2607.02771v1
• Date: Thu, 02 Jul 2026 21:09:13 GMT
• Title: Automated Data Readiness for Scientific AI
• Abstract summary: We present REDI, an open-source framework for transforming scientific datasets into AI-ready data.<n>We show near-ideal parallel scaling to 100 nodes on Frontier for the climate case.<n>These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.

Related papers

• DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams [48.434767119224375]
  We propose a paradigm shift towards Agentic Data Tailoring.<n>To overcome the data scarcity bottleneck in training such high-order capabilities, we design a two-stage pipeline.<n>$textDataClaw_0$-9B model synergizes Supervised Fine-Tuning with Group Relative Policy Optimization.
  arXiv  Detail & Related papers  (2026-06-19T11:31:43Z)
• DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving [55.85201665674226]
  High-quality training data is essential to large language models (LLMs)<n>Existing automatic data preparation methods rely on predefined pipelines or customized human instructions.<n>We introduce DataEvolver, the first self-evolving data preparation system that automatically constructs pipelines to transform raw data into high-quality data.
  arXiv  Detail & Related papers  (2026-06-05T07:44:34Z)
• kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation [0.5833117322405447]
  We introduce kRAIG, an AI agent that translates natural language specifications into production-ready KubeReason Pipelines (KFP)<n>To resolve ambiguity in user intent, we propose ReQuesAct, an interaction framework that explicitly clarifies intent prior to pipeline synthesis.<n>Our framework achieves a 3x improvement in extraction and loading success and a 25 percent increase in transformation accuracy compared to state-of-the-art agentic baselines.
  arXiv  Detail & Related papers  (2026-03-19T17:53:00Z)
• Dataforge: A Data Agent Platform for Autonomous Data Engineering [22.691284342164334]
  Data Agent is a fully autonomous system specialized for tabular data.<n>It automatically performs data cleaning, hierarchical routing, and feature-level optimization through dual feedback loops.<n>It embodies three core principles: automatic, safe, and non-expert friendly, which ensure end-to-end reliability without human supervision.
  arXiv  Detail & Related papers  (2025-11-09T01:58:13Z)
• Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents [85.02904078131682]
  We introduce the agent data protocol (ADP), a light-weight representation language that serves as an "interlingua" between agent datasets.<n> ADP is expressive enough to capture a large variety of tasks, including API/tool use, browsing, coding, software engineering, and general agentic.<n>All code and data are released publicly, in the hope that ADP could help lower the barrier to standardized, scalable, and reproducible agent training.
  arXiv  Detail & Related papers  (2025-10-28T17:53:13Z)
• A Survey of Data Agents: Emerging Paradigm or Overstated Hype? [66.1526688475023]
  "Data agent" currently suffers from terminological ambiguity and inconsistent adoption.<n>This survey introduces the first systematic hierarchical taxonomy for data agents.<n>We conclude with a forward-looking roadmap, envisioning the advent of proactive, generative data agents.
  arXiv  Detail & Related papers  (2025-10-27T17:54:07Z)
• Data Readiness for Scientific AI at Scale [0.0]
  This paper examines how Data Readiness for AI (DRAI) principles apply to leadership-scale scientific datasets used to train foundation models.<n>We analyze archetypal across four representative domains - climate, nuclear fusion, bio/health, and materials.<n>We introduce a two-dimensional readiness framework composed of Data Readiness Levels (raw to AI-ready) and Data Processing Stages (ingest to shard)
  arXiv  Detail & Related papers  (2025-07-30T18:30:37Z)
• Text embedding models can be great data engineers [0.0]
  We propose ADEPT, an automated data engineering pipeline via text embeddings.<n>We show that ADEPT outperforms the best existing benchmarks in a diverse set of datasets.
  arXiv  Detail & Related papers  (2025-05-20T18:12:19Z)
• Zero-shot Composed Text-Image Retrieval [72.43790281036584]
  We consider the problem of composed image retrieval (CIR) It aims to train a model that can fuse multi-modal information, e.g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
  arXiv  Detail & Related papers  (2023-06-12T17:56:01Z)
• Scalable Modular Synthetic Data Generation for Advancing Aerial Autonomy [2.9005223064604078]
  We introduce a scalable Aerial Synthetic Data Augmentation (ASDA) framework tailored to aerial autonomy applications. ASDA extends a central data collection engine with two scriptable pipelines that automatically perform scene and data augmentations. We demonstrate the effectiveness of our method in automatically generating diverse datasets.
  arXiv  Detail & Related papers  (2022-11-10T04:37:41Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.