Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

• URL: http://arxiv.org/abs/2607.17100v1
• Date: Sun, 19 Jul 2026 07:02:44 GMT
• Title: Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
• Abstract summary: An AI research agent can improve the score it sees without finding a modelling change that works on new materials.<n>We show that closed-loop agents can produce decisions that survive unseen evidence and code changes that can be reused across tasks and combined.
• Score: 13.399139714704226
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An AI research agent can improve the score it sees without finding a modelling change that works on new materials. We ask a stricter question. After repeated experiments, does the selected change survive on data that never entered the loop, and can its code be reused? We separate the search into changes to features, models, representations, and training data. Seven searches produce 701 evaluated changes across ten Matbench endpoints. Agents receive only the mean over five inner folds, reducing reliance on any single development split. We then freeze the selected code and evaluate it once on an untouched holdout. Nine of ten choices remain the best tested single intervention. The surviving changes reveal two materials modelling regimes. With composition alone, feature, model, and representation changes provide comparable routes to improvement. They include held-out MAE reductions of 17.4\% for band gap and 18.6\% for steel strength, as well as gains on both classification endpoints, while screened external data adds little. For structure tasks, richer geometry descriptors and model or calibration changes lower mean held-out MAE by 14.6\% and 7.1\% and lead on different property families, whereas composition embeddings do not transfer. Combining separately found feature and model changes yields a 26.3\% mean held-out improvement. These results show in materials prediction that closed-loop agents can produce decisions that survive unseen evidence and code changes that can be reused across tasks and combined. More broadly, they provide an evaluation design for testing executable discoveries beyond the feedback loop.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists [10.23869027501654]
  We present Upgrade, a decision-driven longitudinal benchmark covering four consecutive Qwen releases.<n>Upgrade disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether assets transfer across versions, and what recovery resources are usable.
  arXiv  Detail & Related papers  (2026-08-21T09:37:47Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generation [0.0]
  We tested four prompt conditions (no persona, two engineer personas, and a research-librarian persona), 12 code-generation tasks, two frontier models, and five runs per cell.<n>Under pre-registered mixed-effects analysis, the condition-by-model interaction was significant for provider-reported output tokens.<n>On Claude Opus, the minimalist engineer persona reduced visible output by 30% without improving correctness, while the thorough engineer persona increased output without a correctness gain.
  arXiv  Detail & Related papers  (2026-07-19T21:47:58Z)
• Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity [10.3661763716032]
  In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights.<n>We present a self-evolving agent-harness framework that separates proposing changes from crediting them.
  arXiv  Detail & Related papers  (2026-07-15T10:26:26Z)
• Contrastive Reflection for Iterative Prompt Optimization [0.6533652962118279]
  We present Contrastive Reflection, an iterative prompt-optimization framework for agentic IR.<n>We instantiate the framework with a tree-based slice selector, but the contribution is the contrastive loop rather than the tree itself.
  arXiv  Detail & Related papers  (2026-06-29T19:16:07Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Tool-Augmented Reward Modeling [58.381678612409]
  We propose a tool-augmented preference modeling approach, named Themis, to address limitations by empowering RMs with access to external environments. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines.
  arXiv  Detail & Related papers  (2023-10-02T09:47:40Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.