Long-term Reproducibility for Neural Architecture Search

• URL: http://arxiv.org/abs/2207.04821v1
• Date: Mon, 11 Jul 2022 12:40:26 GMT
• Title: Long-term Reproducibility for Neural Architecture Search
• Abstract summary: There is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries.
• Score: 4.301276597844756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is a sad reflection of modern academia that code is often ignored after publication -- there is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries. This has a significant impact on reproducibility and general scientific progress. Neural Architecture Search (NAS) is no exception to this, with some prior work in reproducibility. However, we argue that these do not consider long-term reproducibility issues. We therefore propose a checklist for long-term NAS reproducibility. We evaluate our checklist against common NAS approaches along with proposing how we can retrospectively make these approaches more long-term reproducible.

Related papers

• Revibing Code from Papers: Reimplementing HCI Artifacts [31.5513342575475]
  We demonstrate the potential of new agentic AI technologies to revibe interactive software.<n>To measure the success of the approach, we describe a revibeability metric.
  arXiv  Detail & Related papers  (2026-08-01T05:18:25Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNN [24.79659407567392]
  Band-split recurrent neural networks (BSRNN) model is promising in this regard, since it yields close to state-of-the-art results on public datasets.<n>Unfortunately, it is not straightforward to reproduce since its full code is not available.<n>This paper attempts to replicate BSRNN as closely as possible to the original paper through extensive experiments.
  arXiv  Detail & Related papers  (2026-03-10T04:54:53Z)
• The Unreasonable Effectiveness of Open Science in AI: A Replication Study [1.991176263126831]
  A crisis has been reported in science, but the extent to which it affects AI research is not yet fully understood. We performed a systematic replication study including 30 highly cited AI studies relying on original materials when available. Six articles were successfully reproduced, while five were partially reproduced.
  arXiv  Detail & Related papers  (2024-12-20T12:33:27Z)
• CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
  We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation. We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
  arXiv  Detail & Related papers  (2024-06-20T16:59:52Z)
• Shortcomings of Question Answering Based Factuality Frameworks for Error Localization [51.01957350348377]
  We show that question answering (QA)-based factuality metrics fail to correctly identify error spans in generated summaries. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.
  arXiv  Detail & Related papers  (2022-10-13T05:23:38Z)
• RealTime QA: What's the Answer Right Now? [137.04039209995932]
  We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer.
  arXiv  Detail & Related papers  (2022-07-27T07:26:01Z)
• Geodesics, Non-linearities and the Archive of Novelty Search [69.6462706723023]
  We show that a key effect of the archive is that it counterbalances the exploration biases that result from the use of inadequate behavior metrics. Our observations seem to hint that attributing a more active role to the archive in sampling can be beneficial.
  arXiv  Detail & Related papers  (2022-05-06T12:03:40Z)
• Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code [74.28810048824519]
  Repro is an open-source library which aims at improving the usability of research code. It provides a lightweight Python API for running software released by researchers within Docker containers.
  arXiv  Detail & Related papers  (2022-04-29T01:54:54Z)
• A Siren Song of Open Source Reproducibility [36.754180527471895]
  We argue that this focus on code for replication is misguided if we want to improve the state of reproducible research. There is a lack of evidence for effective actions taken by conferences to encourage and reward. We argue that venues must take more action to advance reproducible machine learning research today.
  arXiv  Detail & Related papers  (2022-04-09T03:06:40Z)
• 1-2-3 Reproducibility for Quantum Software Experiments [12.312854439969051]
  We propose a 1-2-3 approach to engineering for quantum software experiments. Our scheme ascertains long-term traceability even when the quantum processor itself is no longer accessible.
  arXiv  Detail & Related papers  (2022-01-28T10:54:34Z)
• Avalanche: an End-to-End Library for Continual Learning [81.84325803942811]
  We propose Avalanche, an open-source library for continual learning research based on PyTorch. Avalanche is designed to provide a shared and collaborative for fast prototyping, training, and reproducible evaluation of continual learning algorithms.
  arXiv  Detail & Related papers  (2021-04-01T11:31:46Z)
• Research Reproducibility as a Survival Analysis [22.66983713481359]
  We consider modeling a paper as a survival analysis problem. We show how a survival analysis allows us to draw new insights that better explain prior longitudinal data.
  arXiv  Detail & Related papers  (2020-12-17T20:56:53Z)
• [not Rp] Reproducibility of 'Poincare dodecahedral space parameter estimates' [0.0]
  This paper attempts to reproduce a cosmic topology observational paper published in 2008. The reproduction steps are defined formally in a free-licensed git repository package "0807.4260"
  arXiv  Detail & Related papers  (2020-07-23T21:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.