Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

• URL: http://arxiv.org/abs/2608.31076v1
• Date: Mon, 31 Aug 2026 16:48:51 GMT
• Title: Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
• Abstract summary: AutoSciRub is an evaluation-first framework that induces a task-specific executable before research execution.<n>AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria.<n>During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report.
• Score: 16.43230671992556
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs [16.098240211046548]
  ARAC-Bench is a Researcher-Mimicking Evaluation framework.<n>It shifts the objective from matching final answers to reproducing high-quality human research processes.<n>ARAC-Bench provides not only a fine-grained diagnostic tool but also a scalable reward signal.
  arXiv  Detail & Related papers  (2026-08-13T03:48:07Z)
• From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution [14.763650534859176]
  We introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories.<n>The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline.<n>Our data, implementation, and results are publicly available.
  arXiv  Detail & Related papers  (2026-08-03T12:43:00Z)
• An Agentic Approach Towards Replication Package Quality Evaluation [1.6950215926321557]
  This paper explores an agentic approach for assessing replication package quality.<n>We consolidate 380 requirements from 34 sources into 51 criteria, of which 31 are operationalized for automated artifact-based evaluation.<n>Preliminary evaluation on five replication packages shows high inter-run consistency of 91.4% and 75.4% correctness.
  arXiv  Detail & Related papers  (2026-06-01T10:00:41Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases [17.35673829214932]
  We introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies.<n>Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database.<n>Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%.
  arXiv  Detail & Related papers  (2026-03-24T03:50:34Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation [56.886936435727854]
  DeepResearchEval is an automated framework for deep research task construction and agentic evaluation.<n>For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles.<n>For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
  arXiv  Detail & Related papers  (2026-01-14T18:38:31Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite [75.58737079136942]
  We present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research.<n>Our suite comes with the first scientific research environment with production-grade search tools.<n>Our evaluation of 57 agents across 22 agent classes reveals several interesting findings.
  arXiv  Detail & Related papers  (2025-10-24T17:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.