Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents

• URL: http://arxiv.org/abs/2606.27416v1
• Date: Thu, 25 Jun 2026 17:52:26 GMT
• Title: Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents
• Abstract summary: We present Glite ARF, an open-source Python framework for running LLM coding agents in parallel on a research repository.<n>The framework defines a three-role stack: a human researcher chooses which hypotheses to test, coding agents implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation.<n>Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: LLM coding agents make it tempting to automate empirical research by delegating experiments to them directly, but naive delegation does not scale to large projects: low-rate instruction lapses compound into broken, irreproducible artefacts. To address this problem, we present Glite ARF, an open-source Python framework for running many LLM coding agents in parallel on a research repository without sacrificing reproducibility or auditability. The framework defines a three-role stack: a human researcher chooses which hypotheses to test, coding agents (Claude Code, Codex CLI) implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation, immutability of completed work, a corrections overlay, and a materialised project overview. We call this verifier-driven research: the rules of the research process live in code that fails loudly when violated, not in prose that agents are merely asked to follow. Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages (Spanish, German, Mandarin) and reducing the official baseline RMSE by 29.9% (closed) and 35.9% (open). The campaign comprised 273 tracked tasks (146 experiment runs) across 129 feature sets, run by up to twelve parallel agents orchestrated from a single laptop - with some model training on rented A100s - at approximately \$450 in LLM API spend (\$498 total third-party cost), and structured per-fold provenance let us catch and strip four target-leaking feature sets, correcting an implausible 0.609 RMSE to 0.802. Across three campaigns in three domains, the framework's structural machinery adds only about 1% of wall-clock time. Framework and a public demo project accompany this paper.

Related papers

• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves [1.0312968200748116]
  Large language models can generate substantial code and draft research text, but research-software projects require more than either artifact alone.<n>We identify two LM-specific failure modes: hallucination accumulation and desynchronization.<n>We propose Comet-H, an iterative prompt that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coordinates of a single workspace state.
  arXiv  Detail & Related papers  (2026-04-29T21:28:17Z)
• The cognitive companion: a lightweight parallel monitoring architecture for detecting and recovering from reasoning degradation in LLM agents [0.0]
  Large language model (LLM) agents on multi-step tasks suffer reasoning degradation, looping, drift, stuck states, at rates up to 30% on hard tasks.<n>This paper introduces the Cognitive Companion, a parallel monitoring architecture with two implementations.
  arXiv  Detail & Related papers  (2026-04-15T11:44:20Z)
• AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits [14.213358505741105]
  AgentSZZ is an agent-based framework for identifying bug-inducing commits.<n>Unlike prior methods, AgentSZZ integrates task-specific tools, domain knowledge, and a ReAct-style loop to enable adaptive and causal tracing of bugs.<n>Experiments show that AgentSZZ consistently outperforms state-of-the-art SZZ algorithms across all settings.
  arXiv  Detail & Related papers  (2026-04-03T02:54:02Z)
• Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure [0.0]
  Coding agents represent a new paradigm in automated software engineering, combining the reasoning capabilities of Large Language Models with tool-augmented interaction loops.<n>Top-ranked LLM-based coding agents still fail on over 20% of benchmarked problems.<n>We present a large-scale empirical study analyzing 9,374 trajectories from 19 agents on 500 tasks.
  arXiv  Detail & Related papers  (2026-04-02T21:56:23Z)
• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization [54.965787768076254]
  Large Language Models have been recently exploited as judges for complex natural language processing tasks, such as Q&A.<n>We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization.
  arXiv  Detail & Related papers  (2025-07-22T13:40:26Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• Towards Exception Safety Code Generation with Intermediate Representation Agents Framework [54.03528377384397]
  Large Language Models (LLMs) often struggle with robust exception handling in generated code, leading to fragile programs that are prone to runtime errors.<n>We propose Seeker, a novel multi-agent framework that enforces exception safety in LLM generated code through an Intermediate Representation (IR) approach.<n>Seeker decomposes exception handling into five specialized agents: Scanner, Detector, Predator, Ranker, and Handler.
  arXiv  Detail & Related papers  (2024-10-09T14:45:45Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• AgentFL: Scaling LLM-based Fault Localization to Project-Level Context [11.147750199280813]
  This paper presents AgentFL, a multi-agent system based on ChatGPT for automated fault localization.<n>By simulating the behavior of a human developer, AgentFL models the FL task as a three-step process, which involves comprehension, navigation, and confirmation.<n>The evaluation on the widely used Defects4J-V1.2.0 benchmark shows that AgentFL can localize 157 out of 395 bugs within Top-1.
  arXiv  Detail & Related papers  (2024-03-25T01:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.