PaperGym: Rubric-Centered Evolution for Research-Plan Generation

• URL: http://arxiv.org/abs/2608.31119v1
• Date: Mon, 31 Aug 2026 17:31:18 GMT
• Title: PaperGym: Rubric-Centered Evolution for Research-Plan Generation
• Abstract summary: We introduce PaperGym, a unified framework that turns each research paper into a complete training environment.<n>PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments.<n>With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% forHub Science.
• Score: 69.5921878728483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs [0.0]
  Large language models are being proposed as agents in scientific, in domains where no downstream verifier exists.<n>We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing.<n>The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment.
  arXiv  Detail & Related papers  (2026-08-11T20:30:32Z)
• ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research [142.29356526274387]
  We present ResearchClawBench, a benchmark for evaluating autonomous scientific research.<n>Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation.
  arXiv  Detail & Related papers  (2026-05-28T16:27:40Z)
• How Far Are We From True Auto-Research? [20.195549933333222]
  ResearchArena is a minimal scaffold that lets off-the-shelf agents carry out the full research loop themselves.<n>Across 13 computer science seeds and 3 trials per agent-domain pair, ResearchArena yields 117 agent-generated papers.<n>None of the 117 agent-generated papers reaches the acceptance bar of a top-tier venue.
  arXiv  Detail & Related papers  (2026-05-18T22:20:33Z)
• The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
  We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
  arXiv  Detail & Related papers  (2026-04-27T16:23:09Z)
• ResearchGym: Evaluating Language Model Agents on Real-World AI Research [48.46915933681714]
  We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research.<n>To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL.<n>In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap.
  arXiv  Detail & Related papers  (2026-02-16T19:00:03Z)
• Can Deep Research Agents Find and Organize? Evaluating the Synthesis Gap with Expert Taxonomies [57.11324429385405]
  We introduce TaxoBench, a diagnostic benchmark derived from 72 computer science surveys.<n>We manually extract expert-authored taxonomy trees containing 3,815 precisely categorized citations as ground truth.<n>Best agent recalls only 20.9% of expert-selected papers, and even with perfect input, the best model achieves only 0.31 ARI in organization.
  arXiv  Detail & Related papers  (2026-01-18T11:57:09Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• PaperBench: Evaluating AI's Ability to Replicate AI Research [3.4567792239799133]
  PaperBench is a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research.<n>Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch.<n>PaperBench contains 8,316 individually gradable tasks.
  arXiv  Detail & Related papers  (2025-04-02T15:55:24Z)
• Generation of Highlights from Research Papers Using Pointer-Generator Networks and SciBERT Embeddings [5.095525589147811]
  We use a pointer-generator network with coverage mechanism and a contextual embedding layer at the input that encodes the input tokens into SciBERT embeddings. We test our model on a benchmark dataset, CSPubSum, and also present MixSub, a new multi-disciplinary corpus of papers for automatic research highlight generation.
  arXiv  Detail & Related papers  (2023-02-14T12:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.