A Siren Song of Open Source Reproducibility

• URL: http://arxiv.org/abs/2204.04372v1
• Date: Sat, 9 Apr 2022 03:06:40 GMT
• Title: A Siren Song of Open Source Reproducibility
• Abstract summary: We argue that this focus on code for replication is misguided if we want to improve the state of reproducible research. There is a lack of evidence for effective actions taken by conferences to encourage and reward. We argue that venues must take more action to advance reproducible machine learning research today.
• Score: 36.754180527471895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As reproducibility becomes a greater concern, conferences have largely converged to a strategy of asking reviewers to indicate whether code was attached to a submission. This is part of a larger trend of taking action based on assumed ideals, without studying if those actions will yield the desired outcome. Our argument is that this focus on code for replication is misguided if we want to improve the state of reproducible research. This focus can be harmful -- we should not force code to be submitted. There is a lack of evidence for effective actions taken by conferences to encourage and reward reproducibility. We argue that venues must take more action to advance reproducible machine learning research today.

Related papers

• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• Dynamic Rollout Editing for Reducing Overthinking in RL-Trained Reasoning Models [102.76983747945836]
  Long-form chain-of-thought reasoning can improve performance on complex tasks.<n>But models often continue generating unnecessary reasoning after a correct answer has emerged.<n>We study this phenomenon from the perspective of GRPO-style reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-16T13:10:30Z)
• The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNN [24.79659407567392]
  Band-split recurrent neural networks (BSRNN) model is promising in this regard, since it yields close to state-of-the-art results on public datasets.<n>Unfortunately, it is not straightforward to reproduce since its full code is not available.<n>This paper attempts to replicate BSRNN as closely as possible to the original paper through extensive experiments.
  arXiv  Detail & Related papers  (2026-03-10T04:54:53Z)
• Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings [78.05609552686053]
  This work focuses on an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within the semantics.<n>We introduce a new evaluation dataset in Chinese, named CapRetrieval, whose passages are image captions, and queries are phrases inquiring entities or events in various forms.<n>Zero-shot evaluation suggests that encoders may fail on these fine-grained matching, regardless of training sources or model sizes.
  arXiv  Detail & Related papers  (2025-06-10T09:00:33Z)
• Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers [1.4841630983274845]
  Lack of transparency, data or code, poor adherence to standards, and sensitivity of ML training mean that many papers are not even reproducible in principle. Experiments have found worryingly low degrees of similarity with original results. Poor integrity threatens trust in and integrity of research results.
  arXiv  Detail & Related papers  (2024-06-20T13:56:42Z)
• Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning [69.19840497497503]
  It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents. We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2023-09-04T09:09:54Z)
• The Risks of Recourse in Binary Classification [10.067421338825545]
  We study whether providing algorithmic recourse is beneficial or harmful at the population level. We find that there are many plausible scenarios in which providing recourse turns out to be harmful. We conclude that the current concept of algorithmic recourse is not reliably beneficial, and therefore requires rethinking.
  arXiv  Detail & Related papers  (2023-06-01T09:46:43Z)
• Building a Culture of Reproducibility in Academic Research [55.22219308265945]
  Reproducibility is an ideal that no researcher would dispute "in the abstract", but when aspirations meet the cold hard reality of the academic grind, often "loses out" In this essay, I share some personal experiences grappling with how to operationalize while balancing its demands against other priorities.
  arXiv  Detail & Related papers  (2022-12-27T16:03:50Z)
• ReAct: Temporal Action Detection with Relational Queries [84.76646044604055]
  This work aims at advancing temporal action detection (TAD) using an encoder-decoder framework with action queries. We first propose a relational attention mechanism in the decoder, which guides the attention among queries based on their relations. Lastly, we propose to predict the localization quality of each action query at inference in order to distinguish high-quality queries.
  arXiv  Detail & Related papers  (2022-07-14T17:46:37Z)
• CGAR: Critic Guided Action Redistribution in Reinforcement Leaning [27.8704943323583]
  This paper introduces the novel Critic Guided Action Redistribution (CGAR) algorithm and tests it on the OpenAI MuJoCo tasks. Under the settings of the off-policy actor critic algorithms, we demonstrate that the critic can bring more expected discounted rewards than or at least equal to the actor.
  arXiv  Detail & Related papers  (2022-06-23T06:33:14Z)
• Does the Market of Citations Reward Reproducible Work? [36.754180527471895]
  We show that certain fields of study such as Medicine and Machine Learning (ML) do correlate reproducible works with more citations. We find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations.
  arXiv  Detail & Related papers  (2022-04-08T04:03:17Z)
• Improving Experience Replay with Successor Representation [0.0]
  Prioritized experience replay is a reinforcement learning technique shown to speed up learning. Recent work in neuroscience suggests that, in biological organisms, replay is prioritized by both gain and need.
  arXiv  Detail & Related papers  (2021-11-29T05:25:54Z)
• Offline Reinforcement Learning as Anti-Exploration [49.72457136766916]
  We take inspiration from the literature on bonus-based exploration to design a new offline RL agent. The core idea is to subtract a prediction-based exploration bonus from the reward, instead of adding it for exploration. We show that our agent is competitive with the state of the art on a set of continuous control locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2021-06-11T14:41:30Z)
• Empirically Verifying Hypotheses Using Reinforcement Learning [58.09414653169534]
  This paper formulates hypothesis verification as an RL problem. We aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to generate observations which can help predict whether the hypothesis is true or false.
  arXiv  Detail & Related papers  (2020-06-29T01:01:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.