Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves

• URL: http://arxiv.org/abs/2604.27209v2
• Date: Fri, 01 May 2026 15:02:38 GMT
• Title: Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves
• Abstract summary: Large language models can generate substantial code and draft research text, but research-software projects require more than either artifact alone.<n>We identify two LM-specific failure modes: hallucination accumulation and desynchronization.<n>We propose Comet-H, an iterative prompt that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coordinates of a single workspace state.
• Score: 1.0312968200748116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can now generate substantial code and draft research text, but research-software projects require more than either artifact alone. The mathematical thesis, executable system, benchmark surface, and public claims must mature together, yet often drift apart. We identify two LM-specific failure modes: hallucination accumulation, in which claims exceed what code or theory supports and unsupported assertions propagate across sessions; and desynchronization, in which code, theory, or the model's own world model fall out of alignment. We propose Comet-H, an iterative prompt automaton that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coupled coordinates of a single workspace state. At each step, a controller selects the next prompt by scoring it against what the workspace currently lacks, carries unfinished follow-up work forward with a half-life, and re-checks the paper and README against the code and benchmarks whenever documentation changes. We frame prompt selection as a small contextual bandit problem over prompt families, with prompts as arms, workspace deficits as context, and a hand-weighted linear score. This transparent scorer, paired with a fading record of unfinished work, bounds long-horizon follow-ups, requires no learned policy, and makes each prompt choice legible from the workspace. We created a portfolio of 46 research-software repositories across two dozen domains. We study A3 in depth, a Python static-analysis tool built entirely within the loop, which reaches (F1 = 0.768) on a 90-case benchmark, compared with a next-best baseline of 0.364. Across approximately 400 commits, we find that audit-and-contraction passes dominate the later phases of every successful trajectory.

Related papers

• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents [0.0]
  We present Glite ARF, an open-source Python framework for running LLM coding agents in parallel on a research repository.<n>The framework defines a three-role stack: a human researcher chooses which hypotheses to test, coding agents implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation.<n>Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages.
  arXiv  Detail & Related papers  (2026-06-25T17:52:26Z)
• PaperClaw: Harnessing Agents for Autonomous Research and Human-in-the-Loop Refinement [22.280674906144622]
  PAPERCLAW is a harnessed multi-agent system that carries a project autonomously, from a field of study to a finished paper.<n>A full-lifecycle memory keeps each stage in a single living record, so a long run can be paused, inspected, and resumed without losing context.
  arXiv  Detail & Related papers  (2026-06-21T17:37:01Z)
• PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback [0.0]
  PRIMA operates as coordinated multi-agent research systems over multi-hour runs.<n>Main contributions are three operational patterns for surviving failure modes.<n>A Graph Isomorphism case study grounds the architectural claims in a generated artifact.
  arXiv  Detail & Related papers  (2026-05-23T23:27:46Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• Enhancing Understandability and Transparency of Research Software: Tracing Research to Code [1.4499356176178066]
  We propose an LLM-based automation tool that takes in a paper and the software implementing the paper, and generates a trace mapping between research ideas and their locations in code.<n>Initial experiments have shown that the tool can generate quite useful mappings.
  arXiv  Detail & Related papers  (2026-04-12T19:53:41Z)
• Training-Free Acceleration for Document Parsing Vision-Language Model with Hierarchical Speculative Decoding [102.88996030431662]
  We propose a training-free and highly efficient acceleration method for document parsing tasks.<n>Inspired by speculative decoding, we employ a lightweight document parsing pipeline as a draft model to predict batches of future tokens.<n>We demonstrate the effectiveness of our approach on the general-purpose OmniDocBench.
  arXiv  Detail & Related papers  (2026-02-13T14:22:10Z)
• Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysis [0.0]
  We present Datarus-R1-14B, a language model fine-tuned from Qwen 2.5-14B-Instruct to act as a virtual data analyst and graduate-level problem solver.<n>Datarus is trained not on isolated question-answer pairs but on full analytical trajectories including reasoning steps, code execution, error traces, self-corrections, and final conclusions.
  arXiv  Detail & Related papers  (2025-08-18T21:58:18Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning [70.04746094652653]
  We introduce PaperCoder, a framework that transforms machine learning papers into functional code repositories.<n>PaperCoder operates in three stages: planning, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files.<n>We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations.
  arXiv  Detail & Related papers  (2025-04-24T01:57:01Z)
• Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios [42.75968139336785]
  In large-scale software systems, there are often no fully-fledged bug reports with human-written descriptions when an error occurs.<n>In this case, developers rely on stack traces, i.e., series of function calls that led to the error.<n>Recent works have proposed powerful deep learning-based approaches for this, but they are evaluated and compared in isolation from real-life categories.
  arXiv  Detail & Related papers  (2024-12-19T12:48:17Z)
• Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction [10.428174043080622]
  Large language models are prominently used in real-world applications, often tasked with reasoning over large volumes of documents. We propose SWiM, an evaluation framework that addresses the limitations of standard tests. We also propose medoid voting, a simple, but effective training-free approach that helps alleviate this effect.
  arXiv  Detail & Related papers  (2024-07-04T05:46:20Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.