Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

• URL: http://arxiv.org/abs/2605.25172v1
• Date: Sun, 24 May 2026 17:06:47 GMT
• Title: Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
• Abstract summary: This article is the rejoinder to The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review''<n>We organize our response around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.
• Score: 49.43514488610211
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This article is the rejoinder to ``The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review,'' to appear in the Journal of the American Statistical Association with discussion. To address the practical and theoretical points raised by the discussants, we organize our response around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.

Related papers

• AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality [39.33746637947531]
  We survey reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals.<n>We evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset.
  arXiv  Detail & Related papers  (2026-08-04T12:34:47Z)
• Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering [4.411658619208916]
  Large language models (LLMs) that underpin Agentic AI often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines.<n>This study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, and ISSTA 2025.<n>The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and potential future studies.
  arXiv  Detail & Related papers  (2026-04-01T22:24:08Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• Detecting AI-Generated Content in Academic Peer Reviews [9.46916526949695]
  This study examines the temporal emergence of AI-generated content in peer reviews by applying a detection model trained on historical reviews.<n>We observe minimal detection of AI-generated content before 2022, followed by a substantial increase through 2025.
  arXiv  Detail & Related papers  (2026-01-30T21:15:08Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• ReviewEval: An Evaluation Framework for AI-Generated Reviews [9.35023998408983]
  The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review.<n>We propose ReviewEval, a comprehensive evaluation framework for AI-generated reviews that measures alignment with human assessments, verifies factual accuracy, assesses analytical depth, identifies degree of constructiveness and adherence to reviewer guidelines.<n>This paper establishes essential metrics for AIbased peer review and substantially enhances the reliability and impact of AI-generated reviews in academic research.
  arXiv  Detail & Related papers  (2025-02-17T12:22:11Z)
• The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review [49.43514488610211]
  Author-provided rankings could be leveraged to improve peer review processes at machine learning conferences.<n>We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings.<n>We propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review.
  arXiv  Detail & Related papers  (2024-08-24T01:51:23Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.