Revisiting the Replication Study Design Used in Computing Education Research

• URL: http://arxiv.org/abs/2603.27461v1
• Date: Sun, 29 Mar 2026 00:52:28 GMT
• Title: Revisiting the Replication Study Design Used in Computing Education Research
• Abstract summary: This study aims to examine how the landscape of replication research in Computing Education Research has evolved since 2019.<n>Our SLR identified 63 replication studies among 2,516 published papers.<n>Despite increased discourse around open science and research rigour, the adoption of replication studies in CER has not substantially grown.
• Score: 3.0513214438320078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Replication studies play an important role in Computing Education Research (CER) by supporting the development of consistent and reliable scientific knowledge. However, prior research indicates that the CER community tends to prioritise novel contributions over replication. A 2019 SLR identified only 54 replication studies among 2,269 papers published between 2009 and 2018 across five major CER venues. This study aims to examine how the landscape of replication research in CER has evolved since 2019. Specifically, we investigate whether the prevalence of replication studies has increased and explore current perceptions and experiences of CER researchers regarding replication. We replicated two prior studies. First, we conducted an updated SLR to identify replication studies published between 2019 and 2025 in the same five CER venues. Second, we replicated a survey of Computing Education researchers to better understand their perceptions, experiences, and challenges related to conducting and publishing replication studies. Our SLR identified 63 replication studies among 2,516 published papers. While the proportion of replication studies has increased slightly, overall growth remains limited. We observed a shift toward more published replication studies in journals and an increase in authors replicating their own prior work. Survey results indicate that although many researchers engage in replication within their teaching and research practice, they encounter significant challenges when attempting to publish replication studies. Despite increased discourse around open science and research rigour, the adoption of replication studies in CER has not substantially grown. Our findings offer opportunities for future research to promote replication and to explore how the CER community can encourage researchers to publish replication studies.

Related papers

• Training AI Scientists to Replicate Research [8.913339899077362]
  We develop Replica, a scalable task space for paper replication.<n>We introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality.
  arXiv  Detail & Related papers  (2026-08-13T14:59:27Z)
• DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [152.2148664328137]
  Deep research models perform multi-step research to produce long-form, well-attributed answers.<n>Most open deep research models are trained on short-form QA tasks via reinforcement learning with verifiable rewards.<n>We develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research.
  arXiv  Detail & Related papers  (2025-11-24T18:35:54Z)
• Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies [3.053547151063031]
  We studied 86 articles describing LLM-centric studies, published at ICSE 2024 and 2024.<n>Of the 86 articles, 18 provided research artefacts and used OpenAI models.<n>Of the 18 studies, only five were fit for reproduction. For none of the five studies, we were able to fully reproduce the results.
  arXiv  Detail & Related papers  (2025-10-29T13:31:32Z)
• Hypothesis Hunting with Evolving Networks of Autonomous Scientific Agents [52.50038914857797]
  We term this process hypothesis hunting: the cumulative search for insight through sustained exploration across vast and complex hypothesis spaces.<n>We introduce AScience, a framework modeling discovery as the interaction of agents, networks, and evaluation norms, and implement it as ASCollab.<n> Experiments show that such social dynamics enable the accumulation of expert-rated results along the diversity-quality-novelty frontier.
  arXiv  Detail & Related papers  (2025-10-08T08:47:07Z)
• 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models [58.98176123850354]
  The recent release of DeepSeek-R1 has generated widespread social impact and sparked enthusiasm in the research community for exploring the explicit reasoning paradigm of language models.<n>The implementation details of the released models have not been fully open-sourced by DeepSeek, including DeepSeek-R1-Zero, DeepSeek-R1, and the distilled small models.<n>Many replication studies have emerged aiming to reproduce the strong performance achieved by DeepSeek-R1, reaching comparable performance through similar training procedures and fully open-source data resources.
  arXiv  Detail & Related papers  (2025-05-01T14:28:35Z)
• ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition [67.26124739345332]
  Large language models (LLMs) have demonstrated potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined.<n>We introduce the first large-scale benchmark for evaluating LLMs with a near-sufficient set of sub-tasks of scientific discovery.<n>We develop an automated framework that extracts critical components - research questions, background surveys, inspirations, and hypotheses - from scientific papers.
  arXiv  Detail & Related papers  (2025-03-27T08:09:15Z)
• Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation [58.064940977804596]
  A plethora of new AI models and tools has been proposed, promising to empower researchers and academics worldwide to conduct their research more effectively and efficiently.<n>Ethical concerns regarding shortcomings of these tools and potential for misuse take a particularly prominent place in our discussion.
  arXiv  Detail & Related papers  (2025-02-07T18:26:45Z)
• LLM4SR: A Survey on Large Language Models for Scientific Research [15.533076347375207]
  Large Language Models (LLMs) offer unprecedented support across various stages of the research cycle.<n>This paper presents the first systematic survey dedicated to exploring how LLMs are revolutionizing the scientific research process.
  arXiv  Detail & Related papers  (2025-01-08T06:44:02Z)
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models [56.08917291606421]
  ResearchAgent is an AI-based system for ideation and operationalization of novel work.<n>ResearchAgent automatically defines novel problems, proposes methods and designs experiments, while iteratively refining them.<n>We experimentally validate our ResearchAgent on scientific publications across multiple disciplines.
  arXiv  Detail & Related papers  (2024-04-11T13:36:29Z)
• Automatically Finding and Categorizing Replication Studies [0.0]
  In many fields of experimental science, papers that failed to replicate continue to be cited as a result of the poor discoverability of replication studies. As a first step to creating a system that automatically finds replication studies for a given paper, 334 replication studies and 344 replicated studies were collected.
  arXiv  Detail & Related papers  (2023-11-25T15:27:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.