Enhancing Understandability and Transparency of Research Software: Tracing Research to Code

• URL: http://arxiv.org/abs/2604.10793v1
• Date: Sun, 12 Apr 2026 19:53:41 GMT
• Title: Enhancing Understandability and Transparency of Research Software: Tracing Research to Code
• Abstract summary: We propose an LLM-based automation tool that takes in a paper and the software implementing the paper, and generates a trace mapping between research ideas and their locations in code.<n>Initial experiments have shown that the tool can generate quite useful mappings.
• Score: 1.4499356176178066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between the paper's ideas and the code that implements them. As a time- and staff-saving approach, we propose an LLM-based automation tool that takes in a paper and the software implementing the paper, and generates a trace mapping between research ideas and their locations in code. Initial experiments have shown that the tool can generate quite useful mappings.

Related papers

• VERITAS: Towards a General-Purpose Replication Tool for Scientific Research [19.45546798679836]
  We present VERITAS, a domain-agnostic replication framework built around CLI coding agents.<n>Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs.<n>The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched.<n>We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics.
  arXiv  Detail & Related papers  (2026-07-03T03:55:59Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework [82.66443886385125]
  Paper Circle is a discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature.<n>The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-agent scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, and figures.
  arXiv  Detail & Related papers  (2026-04-07T17:59:58Z)
• PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature [11.804526152911386]
  We propose PaperArena, an evaluation benchmark for large language model (LLM) based agents to address real-world research questions.<n>Given a research question, agents should integrate diverse formats across multiple papers through reasoning and interacting with appropriate tools.<n> Experimental results reveal that even the most advanced LLM powering a well-established agent system achieves merely 38.78% average accuracy.
  arXiv  Detail & Related papers  (2025-10-13T02:10:39Z)
• Rethinking Code Review Workflows with LLM Assistance: An Empirical Study [2.9593087583214173]
  This paper combines an exploratory field study of current code review practices with a field experiment involving two variations of an LLM-assisted code review tool.<n>The study identifies key challenges in traditional code reviews, including frequent context switching and insufficient contextual information.<n>In the field experiment, we developed two prototype variations: one offering LLM-generated reviews upfront and the other enabling on-demand interaction.
  arXiv  Detail & Related papers  (2025-05-22T07:54:07Z)
• Towards Leveraging Large Language Model Summaries for Topic Modeling in Source Code [0.0]
  Large language models (LLMs) have demonstrated remarkable program comprehension capabilities.<n> transformer-based topic modeling techniques offer effective ways to extract semantic information from text.<n>This paper proposes and explores a novel approach that combines these strengths to automatically identify meaningful topics in a corpus of Python programs.
  arXiv  Detail & Related papers  (2025-04-24T10:30:40Z)
• Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning [70.04746094652653]
  We introduce PaperCoder, a framework that transforms machine learning papers into functional code repositories.<n>PaperCoder operates in three stages: planning, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files.<n>We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations.
  arXiv  Detail & Related papers  (2025-04-24T01:57:01Z)
• Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol [83.90769864167301]
  Literature review tables are essential for summarizing and comparing collections of scientific papers.<n>We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers.<n>Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques.
  arXiv  Detail & Related papers  (2025-04-14T14:52:28Z)
• Aligning Multimodal LLM with Human Preference: A Survey [62.89722942008262]
  Large language models (LLMs) can handle a wide variety of general tasks with simple prompts, without the need for task-specific training.<n>Multimodal Large Language Models (MLLMs) have demonstrated impressive potential in tackling complex tasks involving visual, auditory, and textual data.<n>However, critical issues related to truthfulness, safety, o1-like reasoning, and alignment with human preference remain insufficiently addressed.
  arXiv  Detail & Related papers  (2025-03-18T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.