PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress
- URL: http://arxiv.org/abs/2609.16995v1
- Date: Tue, 15 Sep 2026 11:08:11 GMT
- Title: PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress
- Abstract summary: PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
- Score: 130.58520199343707
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Autoresearch agents are reshaping the research ecosystem, but they can also let flawed claims enter the literature at scale. Human advisors catch such issues in drafts through careful, traceable feedback, yet advisor-style assessment requires extensive manual effort and does not scale. To shift automated paper assessment from a judge to a diagnostician, we introduce PaperDoctor, an agent framework for pre-submission feedback with three key innovations. First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments through three layers: L1 surface screening, L2 typed verifiers that route each claim to the appropriate evidence, and L3 reproducers that rerun experiments by priority. Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion, making critiques auditable and actionable. Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget, surfacing reproducibility gaps and quantitative limitations that are invisible from the manuscript alone. We evaluate PaperDoctor on 30 in-progress papers, yielding 70.6% agreement and all positive holistic scores, and on 40 manuscripts across machine learning, natural science, and social science, covering human- and AI-authored papers with code. Overall, PaperDoctor produces more auditable feedback than human and other agentic reviewers, pairs critiques with concrete suggestions by design, and complements dimensions often overlooked by human reviewers. We also develop an interactive interface that lets authors browse findings grounded in their paper. PaperDoctor reframes automated paper assessment as diagnosis rather than verdict, taking a concrete step toward AI advisors for more rigorous AI-assisted scientific discovery.
Related papers
- Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
arXiv Detail & Related papers (2026-08-18T17:29:00Z) - Towards Automating Scientific Review with Google's Paper Assistant Tool [43.64244794415017]
Paper Assistant Tool (PAT) is an agentic AI framework built for deep scientific review and verification.<n>Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences demonstrate its ability to identify critical errors.<n> PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.
arXiv Detail & Related papers (2026-06-26T17:19:17Z) - PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf [51.79108552674703]
We present PaperMentor, a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments.<n>PaperMentor integrates an expert skill library carefully curated from established researchers' writing advice with 12 specialized agents covering different aspects of paper writing.<n>In a user study, 90.6% of the generated comments were rated actionable and 67.5% were rated valid, significantly outperforming a GPT-5.2 baseline uswithout the skill library.
arXiv Detail & Related papers (2026-06-07T22:17:15Z) - When AI reviews science: Can we trust the referee? [73.47745294608072]
We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
arXiv Detail & Related papers (2026-04-26T08:03:32Z) - Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers [41.637398542914816]
This paper introduces the first systematic evaluation framework for quantifying the quality and risks of papers written by modern coding agents.<n>PaperRecon disentangles the evaluation of the AI-written papers into two dimensions, Presentation and Hallucination.<n>For evaluation, we introduce PaperWrite-Bench, a benchmark of 51 papers from top-tier venues across diverse domains published after 2025.
arXiv Detail & Related papers (2026-04-01T16:48:04Z) - Towards a Medical AI Scientist [73.6056699962416]
We introduce Medical AI Scientist, the first autonomous research framework tailored to clinical autonomous research.<n>The framework operates under 3 research modes, namely paper-based reproduction, literature-inspired innovation, and task-driven exploration.<n>Our system achieves strong alignment between the proposed method and its implementation, while also demonstrating significantly higher success rates in executable experiments.
arXiv Detail & Related papers (2026-03-30T15:37:25Z) - APRES: An Agentic Paper Revision and Evaluation System [44.44345338738518]
The primary way scientists communicate their work and receive feedback from the community is through peer review.<n>In this paper, we introduce a novel method APRES powered by Large Language Models (LLMs) to update a scientific papers text based on an evaluation rubric.<n>Our automated method discovers a rubric that is highly predictive of future citation counts, and integrate it with APRES in an automated system that revises papers to enhance their quality and impact.
arXiv Detail & Related papers (2026-03-03T16:29:13Z) - ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review [23.630458187587223]
ReviewerToo is a framework for studying and deploying AI-assisted peer review.<n>It supports systematic experiments with specialized reviewer personas and structured evaluation criteria.<n>We show how AI can enhance consistency, coverage, and fairness while leaving complex evaluative judgments to domain experts.
arXiv Detail & Related papers (2025-10-09T23:53:19Z) - Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
arXiv Detail & Related papers (2025-08-29T08:48:00Z) - A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
arXiv Detail & Related papers (2024-02-20T11:28:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.