Building a research-software catalog with a coding agent: from hackathon prototype to public deployment

• URL: http://arxiv.org/abs/2609.04711v1
• Date: Fri, 04 Sep 2026 04:31:51 GMT
• Title: Building a research-software catalog with a coding agent: from hackathon prototype to public deployment
• Abstract summary: Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance.<n>We develop a repository catalog during a hackathon and examine the engineering required to make it suitable for public deployment.<n>We explore whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackathon and subsequently examined the engineering required to make it suitable for public deployment, including adversarial review, data-quality checks, browser-level validation, and publication safeguards. We then explored whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal, through a retrieval agent under development for MateriApps that combines curated portal metadata, external documentation, vector search, and local language-model generation. Implementation with coding agents was rapid, but achieving reliable operation required substantial additional engineering: the most consequential problems were not crashes but silent failures that produced plausible yet incomplete or incorrect outputs, arising from incomplete data acquisition, misleading assessments, and retrieval or preprocessing failures. These observations suggest that AI-assisted software portals require explicit validation, monitoring, and repeated review, and that curated metadata and maintained documentation remain essential. The MateriApps work is exploratory and remains under active development, so the observations reported for it are preliminary; a comparable combination of curated metadata, automatically collected documentation, and retrieval-based assistance may nevertheless be useful for extending other research-software portals.

Related papers

• From Final Artifacts to Trajectories: Retrospective Process Supervision for Evidence-Grounded Long-Form Generation [49.496216822640974]
  RetroGen is a self-improving framework for retrospective process supervision.<n>It reconstructs latent trajectories from expert artifacts, verifies them against the artifact and supporting evidence, and trains models on their own successful reconstruction data.<n>Experiments show that RetroGen improves grounding, faithful synthesis, and long-form evidence-seeking agent tasks.
  arXiv  Detail & Related papers  (2026-08-31T08:47:39Z)
• Self-Evolving Coding Agents [11.43197414613246]
  Self-evolving coding agents can inspect repositories, invoke tools, execute tests, debug failures, and generate patches.<n>We first define self-evolving coding agents and distinguish them from conventional coding agents and general self-evolving agents.<n>We find that executable feedback, repository-level context, and coding trajectories give software engineering a distinctive role as a natural domain for agent self-evolution.
  arXiv  Detail & Related papers  (2026-08-04T09:43:46Z)
• SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration [77.28003194268977]
  SearchOS is a system-level multi-agent framework that turns fragile, implicit search progress into explicit, persistent, and shared state.<n>Built on SOCM, SearchOS applies a pipeline-parallel scheduling mechanism that overlaps the execution of sub-agents.<n>SearchOS leads all metrics among the evaluated single- and multi-agent baselines.
  arXiv  Detail & Related papers  (2026-07-16T17:51:23Z)
• Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development [0.0]
  Agentic AI coding systems can inspect repositories, plan implementation steps, edit files, run tests, and submit pull requests.<n>Current evidence does not support the simple claim that autonomous code generation automatically improves engineering outcomes.<n>This paper argues that the central problem is no longer prompt engineering; it is engineering process control.
  arXiv  Detail & Related papers  (2026-05-19T20:10:13Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• Does My README File Need To Be Updated? Exploring LLM-Based README Maintenance [13.873979933468268]
  Outdated documentation is perceived by developers as one of the most frequent and severe challenges with gaining project understanding.<n>We propose a lightweight Large Language Model (LLM)-driven approach to facilitate precise, localised file updates within a human-in-the-loop workflow.
  arXiv  Detail & Related papers  (2026-02-28T06:04:45Z)
• Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions [8.97512410819274]
  This paper presents the first empirical study on how state-of-the-art multi-agent systems perform in dataset adaptation tasks.<n>We evaluate GitHub Copilot on adapting SE research artifacts from benchmark repositories including ROCODE and LogHub2.0.<n>Results show that current systems can identify key files and generate partial adaptations but rarely produce correct implementations.
  arXiv  Detail & Related papers  (2025-11-26T13:26:11Z)
• From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents [96.65646344634524]
  Large Language Models (LLMs), endowed with reasoning and agentic capabilities, are ushering in a new paradigm termed Agentic Deep Research.<n>We trace the evolution from static web search to interactive, agent-based systems that plan, explore, and learn.<n>We demonstrate that Agentic Deep Research not only significantly outperforms existing approaches, but is also poised to become the dominant paradigm for future information seeking.
  arXiv  Detail & Related papers  (2025-06-23T17:27:19Z)
• Identity resolution of software metadata using Large Language Models [0.0]
  This article presents an evaluation of instruction-tuned large language models for the task of software metadata identity resolution.<n>We benchmarked multiple models against a human-annotated gold standard, examined their behavior on ambiguous cases, and introduced an agreement-based proxy for high-confidence automated decisions.
  arXiv  Detail & Related papers  (2025-05-29T14:47:31Z)
• R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science [70.1638335489284]
  High-level machine learning engineering tasks remain labor-intensive and iterative.<n>We introduce R&D-Agent, a comprehensive, decoupled, and framework that formalizes the machine learning process.<n>R&D-Agent defines the MLE into two phases and six components, turning agent design for MLE into a principled, testable process.
  arXiv  Detail & Related papers  (2025-05-20T06:07:00Z)
• RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation [79.83270415843857]
  We introduce RepoAgent, a large language model powered open-source framework aimed at proactively generating, maintaining, and updating code documentation. We have validated the effectiveness of our approach, showing that RepoAgent excels in generating high-quality repository-level documentation.
  arXiv  Detail & Related papers  (2024-02-26T15:39:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.