Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

• URL: http://arxiv.org/abs/2604.24819v1
• Date: Mon, 27 Apr 2026 14:05:18 GMT
• Title: Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora
• Abstract summary: We show that a structured knowledge representation extracted from the source corpus serves as the shared foundation for both training data and evaluation.<n>This work establishes a principled foundation for the reliable engineering of human expertise into language models.
• Score: 33.535457603064906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliably transferring specialized human knowledge from text into large language models remains a fundamental challenge in artificial intelligence. Fine-tuning on domain corpora has enabled substantial capability gains, but the process operates without feedback: when a model fails on a domain task, there is no method to diagnose what is deficient in the training data, and the only recourse is to add more data indiscriminately. Here we show that when a structured knowledge representation extracted from the source corpus serves as the shared foundation for both training data and evaluation, the complete data-engineering lifecycle maps onto the software development lifecycle in a precise and operative way: training data becomes source code specifying what the model should learn, model training becomes compilation, benchmarking becomes unit testing, and failure-driven data repair becomes debugging. Under this correspondence, model failures decompose into concept-level gaps and reasoning-chain breaks that can be traced back to specific deficiencies in the data and repaired through targeted patches, with each repair cycle producing consistent improvements across model scales and architectures without degrading general capabilities. We formalize this principle as Programming with Data and instantiate it across sixteen disciplines spanning the natural sciences, engineering, biomedicine, and the social sciences, releasing a structured knowledge base, benchmark suite, and training corpus as open resources. By demonstrating that the relationship between training data and model behaviour is structurally traceable and systematically repairable, this work establishes a principled foundation for the reliable engineering of human expertise into language models.

Related papers

• DataFoundry: Evolving Data Preparators via Recursive Self-Improvement [34.053745857869636]
  textscDataFoundry represents a data preparator as an evolvable runtime specification and instantiates its evolution with a textsc-as-Modules architecture.<n>We evaluate textscDataFoundry on DataPrep-Bench across mathematics, finance, law, and medicine.
  arXiv  Detail & Related papers  (2026-08-30T18:53:24Z)
• Econometrics with Pre-Trained Embeddings for Unstructured Data [0.0]
  Economists often use off-the-shelf pre-trained deep learning models developed by computer scientists to extract embeddings from unstructured data.<n>Despite the popularity of this practice, its theoretical foundations remain limited.<n>We provide sufficient conditions to overcome these difficulties and derive the convergence rate of machine learning models with pre-trained embeddings.
  arXiv  Detail & Related papers  (2026-07-19T18:37:49Z)
• Exploring a New Competency Modeling Process with Large Language Models [0.0]
  This study proposes a new competency modeling process built on large language models (LLMs)<n> Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data.<n>We introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals.
  arXiv  Detail & Related papers  (2026-02-13T16:46:51Z)
• A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring) [0.0]
  We introduce a model-free framework using surprisal (information theoretic uncertainty) to analyze and perform inferences from raw data.<n>It eliminates distribution modeling, reducing bias, and enabling efficient updates including direct edits and deletion of training data.<n>It emphasizes traceability, interpretability, and data-driven decision making, offering a unified, human-understandable framework for machine learning.
  arXiv  Detail & Related papers  (2025-10-26T19:45:25Z)
• Predict Training Data Quality via Its Geometry in Metric Space [7.056460460498077]
  We propose that the richness of representation and the elimination of redundancy within training data critically influence learning outcomes.<n>To investigate this, we employ persistent homology to extract topological features from data within a metric space.<n>Our findings highlight persistent homology as a powerful tool for analyzing and enhancing the training data that drives AI systems.
  arXiv  Detail & Related papers  (2025-10-12T16:59:28Z)
• Federated Graph Unlearning [23.00839112398916]
  The demand for data privacy has led to the development of frameworks like Federated Graph Learning.<n>The proposed framework employs a bifurcated strategy tailored to the specific unlearning request.<n>The framework achieves substantial improvements in model prediction accuracy across both client and meta-unlearning scenarios.
  arXiv  Detail & Related papers  (2025-08-04T14:57:03Z)
• On Reconstructing Training Data From Bayesian Posteriors and Trained Models [4.061135251278187]
  Publicly releasing the specification of a model with its trained parameters means an adversary can attempt to reconstruct information about the training data.<n>This paper makes three primary contributions: establishing a mathematical framework to express the problem, characterising the features of the training data that are vulnerable via a maximum mean discrepancy equivalance and outlining a score matching framework for reconstructing data in both Bayesian and non-Bayesian models.
  arXiv  Detail & Related papers  (2025-07-24T12:49:41Z)
• Re:Form -- Reducing Human Priors in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny [78.1575956773948]
  Large Language Models (LLMs) trained with Reinforcement Learning (RL) face a significant challenge: their verification processes are neither reliable nor scalable.<n>A promising yet largely uncharted alternative is formal language-based reasoning.<n>Grounding LLMs in rigorous formal systems where generative models operate in formal language spaces (e.g., Dafny) enables the automatic and mathematically provable verification of their reasoning processes and outcomes.
  arXiv  Detail & Related papers  (2025-07-22T08:13:01Z)
• RESTOR: Knowledge Recovery in Machine Unlearning [71.75834077528305]
  Large language models trained on web-scale corpora can contain private or sensitive information.<n>Several machine unlearning algorithms have been proposed to eliminate the effect of such datapoints.<n>We propose the RESTOR framework for machine unlearning evaluation.
  arXiv  Detail & Related papers  (2024-10-31T20:54:35Z)
• Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models [52.40798352740857]
  We introduce the Iterative Contrastive Unlearning (ICU) framework, which consists of three core components.<n>A Knowledge Unlearning Induction module targets specific knowledge for removal using an unlearning loss.<n>A Contrastive Learning Enhancement module preserves the model's expressive capabilities against the pure unlearning goal.<n>An Iterative Unlearning Refinement module dynamically adjusts the unlearning process through ongoing evaluation and updates.
  arXiv  Detail & Related papers  (2024-07-25T07:09:35Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
  We propose a fully decentralized approach, which allows to share knowledge between trained models. Students are trained on the output of their teachers via synthetically generated input data. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
  arXiv  Detail & Related papers  (2021-02-01T14:38:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.