Metag: A dataset to build agentic meta-reviewing capabilities

• URL: http://arxiv.org/abs/2608.20488v2
• Date: Mon, 24 Aug 2026 21:53:47 GMT
• Title: Metag: A dataset to build agentic meta-reviewing capabilities
• Abstract summary: This paper introduces Metag, a dataset to identify changes made to scientific articles during the review-rebuttal process.<n>The resulting dataset consists of 349 high-quality action items tied to paper differences.
• Score: 9.89538433423377
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same time, the continuing growth in conference submissions has increased the burden on meta-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta-reviewing agents, specifically to identify changes made to scientific articles during the review-rebuttal process. Each instance contains a reviewer concern, the author's proposed resolution, and the manuscript diffs implementing the stated change. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions. The resulting dataset consists of 349 high-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review. The dataset is publicly available at https://github.com/microsoft/Metag-dataset.

Related papers

• Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future [50.89292049215399]
  Large language models (LLMs) have motivated methods that assist or automate different stages of the peer review process.<n>We synthesize techniques for (i) peer review generation, including fine-tuning strategies, agent-based systems, RL-based methods, and emerging paradigms to enhance generation.<n>We catalog datasets, compare modeling choices, and discuss limitations, ethical concerns, and future directions.
  arXiv  Detail & Related papers  (2026-04-30T14:28:21Z)
• ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents [50.27474750319121]
  Review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages.
  arXiv  Detail & Related papers  (2026-04-15T16:33:04Z)
• FMMD: A multimodal open peer review dataset based on F1000Research [2.375184015411392]
  FMMD is a multimodal and multidisciplinary open peer review dataset curated from F1000Research.<n>It bridges the current gap by integrating manuscript-level visual and structural data with version-specific reviewer reports and editorial decisions.<n>It provides a comprehensive empirical resource for the development of peer review research.
  arXiv  Detail & Related papers  (2026-02-15T19:36:05Z)
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing [53.85037373860246]
  We introduce Deep Synth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities.<n>We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization)<n>Our results demonstrate that agentic plan-and-write significantly outperform single-turn generation.
  arXiv  Detail & Related papers  (2026-01-07T03:07:52Z)
• What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review [2.9282248958475345]
  We present a large-scale empirical study of ICLR 2017-2025, encompassing over 28,000 submissions.<n>Our results show that factors beyond scientific novelty significantly shape acceptance outcomes.<n>We propose data-driven guidelines for authors, reviewers, and meta-reviewers to enhance transparency and fairness in peer review.
  arXiv  Detail & Related papers  (2025-09-30T03:00:10Z)
• Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue [61.0689885044492]
  We explore the challenges for realizing dialog agents that can effectively assist meta-reviewers.<n>We first address the issue of data scarcity for training dialogue agents.<n>We utilize this data to train dialogue agents tailored for meta-reviewing.
  arXiv  Detail & Related papers  (2025-08-07T11:27:43Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Deep Transfer Learning Based Peer Review Aggregation and Meta-review Generation for Scientific Articles [2.0778556166772986]
  We address two peer review aggregation challenges: paper acceptance decision-making and meta-review generation. Firstly, we propose to automate the process of acceptance decision prediction by applying traditional machine learning algorithms. For the meta-review generation, we propose a transfer learning model based on the T5 model.
  arXiv  Detail & Related papers  (2024-10-05T15:40:37Z)
• CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions [7.503795054002406]
  We propose an original textual resource on the revision step of the writing process of scientific articles. This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews.
  arXiv  Detail & Related papers  (2024-03-01T03:07:32Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation [55.00687185394986]
  We propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews. We introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences. Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions.
  arXiv  Detail & Related papers  (2023-05-24T02:33:35Z)
• MOPRD: A multidisciplinary open peer review dataset [12.808751859133064]
  Open peer review is a growing trend in academic publications. Most of the existing peer review datasets do not provide data that cover the whole peer review process. We construct MOPRD, a multidisciplinary open peer review dataset.
  arXiv  Detail & Related papers  (2022-12-09T16:35:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.