From Discussion to Execution: Replicating Buggy and Correct Data Science Code

• URL: http://arxiv.org/abs/2607.16569v1
• Date: Sat, 18 Jul 2026 00:48:26 GMT
• Title: From Discussion to Execution: Replicating Buggy and Correct Data Science Code
• Abstract summary: We introduce Reprodgen, a framework for replicating executable buggy and patched data science programs from Q&A forum posts.<n>Given a question and its corresponding answer, Reprodgen reconstructs the buggy behavior described in the question and the intended fix described in the answer.<n>We evaluate Reprodgen on Stack Overflow (SO) and GitHub Issues (GI) across seven data science libraries.
• Score: 3.2371089062298317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reproducing reliable data science code from informal sources is challenging due to ambiguous problem specifications, missing dependencies, and performance bottlenecks. Although developer Q&A forums provide rich discussions on diagnosing and fixing real-world issues, the information is often incomplete and unstructured, limiting its use for automated debugging and verification. In this paper, we introduce Reprodgen, a large language model (LLM) based framework for automatically replicating executable buggy and patched data science programs from Q&A forum posts. Given a question and its corresponding answer, Reprodgen reconstructs the buggy behavior described in the question and the intended fix described in the answer, producing executable buggy and patched code pairs that reflect the original discussion. The framework builds structured representations of code intent (CI), functional requirements (FR), and Structured Chain of Thought (SCoT), and iteratively refines code using an LLM-based reviewer until it is executable and semantically consistent. We evaluate Reprodgen on Stack Overflow (SO) and GitHub Issues (GI) across seven data science libraries, including pandas, numpy, and scikit-learn, and construct a benchmark of runnable buggy and patched programs validated by human experts. Our pipeline uses LLMs for semantic assessment, while executability is verified through actual execution. Results show reliable replication with clear differences in model performance.

Related papers

• RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists [12.810164698494281]
  RepoProbe is a novel benchmark for evaluating repository-level code understanding.<n>We propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts.
  arXiv  Detail & Related papers  (2026-08-05T12:49:36Z)
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• VERITAS: Towards a General-Purpose Replication Tool for Scientific Research [19.45546798679836]
  We present VERITAS, a domain-agnostic replication framework built around CLI coding agents.<n>Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs.<n>The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched.<n>We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics.
  arXiv  Detail & Related papers  (2026-07-03T03:55:59Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation [26.17880287280065]
  pyrag is a framework that reformulates multi-hop RAG as program synthesis and execution.<n>pyrag consistently outperforms strong baselines under both training-free and RL-trained settings.
  arXiv  Detail & Related papers  (2026-05-13T04:14:13Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation [1.564663326217051]
  RAGVUE is a framework for evaluation of Retrieval-Augmented Generation (RAG) systems.<n>It decomposes RAG behavior into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration.<n> RAGVUE supports both manual metric selection and fully automated agentic evaluation.
  arXiv  Detail & Related papers  (2025-12-03T07:42:49Z)
• Repeton: Structured Bug Repair with ReAct-Guided Patch-and-Test Cycles [1.387448620257867]
  Large Language Models (LLMs) have shown strong capabilities in code generation and comprehension, yet their application to complex software engineering tasks often suffers from low precision and limited interpretability.<n>We present Repeton, a fully open-source framework that leverages LLMs for precise and automated code manipulation in real-world Git.
  arXiv  Detail & Related papers  (2025-06-09T19:36:40Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• Fact-Checking Complex Claims with Program-Guided Reasoning [99.7212240712869]
  Program-Guided Fact-Checking (ProgramFC) is a novel fact-checking model that decomposes complex claims into simpler sub-tasks. We first leverage the in-context learning ability of large language models to generate reasoning programs. We execute the program by delegating each sub-task to the corresponding sub-task handler.
  arXiv  Detail & Related papers  (2023-05-22T06:11:15Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• FixEval: Execution-based Evaluation of Program Fixes for Programming Problems [23.987104440395576]
  We introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately.
  arXiv  Detail & Related papers  (2022-06-15T20:18:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.