Related papers: Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

URL: http://arxiv.org/abs/2601.15249v2
Date: Thu, 22 Jan 2026 15:51:15 GMT
Title: Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism
Authors: Garrett G. Wen, Buxin Su, Natalie Collina, Zhun Deng, Weijie Su,
Abstract summary: We introduce an author-assisted mechanism to facilitate the selection of best paper awards.<n>Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions.<n>We prove that truthfulness holds even when the utility function is merely nondecreasing and additive.
Score: 10.746401441903174
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.

Related papers

Navigating Through Paper Flood: Advancing LLM-based Paper Evaluation through Domain-Aware Retrieval and Latent Reasoning [30.92327406304362]
We present PaperEval, a novel framework for automated paper evaluation using Large Language Models (LLMs)<n>PaperEval has two key components: 1) a domain-aware paper retrieval module that retrieves relevant concurrent work to support contextualized assessments of novelty and contributions, and 2) a latent reasoning mechanism that enables deep understanding of complex motivations and methodologies.<n> Experiments on two datasets demonstrate that PaperEval consistently outperforms existing methods in both academic impact and paper quality evaluation.
arXiv Detail & Related papers (2025-08-07T08:08:13Z)
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
arXiv Detail & Related papers (2025-06-09T18:37:14Z)
The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review [49.43514488610211]
Author-provided rankings could be leveraged to improve peer review processes at machine learning conferences.<n>We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings.<n>We propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review.
arXiv Detail & Related papers (2024-08-24T01:51:23Z)
From Words to Worth: Newborn Article Impact Prediction with LLM [69.41680520058418]
This paper introduces a promising approach, leveraging the capabilities of LLMs to predict the future impact of newborn articles.<n>The proposed method employs LLM to discern the shared semantic features of highly impactful papers from a large collection of title-abstract pairs.<n>The quantitative results, with an MAE of 0.216 and an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance.
arXiv Detail & Related papers (2024-08-07T17:52:02Z)
EFaR 2023: Efficient Face Recognition Competition [51.77649060180531]
The paper presents the summary of the Efficient Face Recognition Competition (EFaR) held at the 2023 International Joint Conference on Biometrics (IJCB 2023) The competition received 17 submissions from 6 different teams. The submitted solutions are ranked based on a weighted score of the achieved verification accuracies on a diverse set of benchmarks, as well as the deployability given by the number of floating-point operations and model size.
arXiv Detail & Related papers (2023-08-08T09:58:22Z)
Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment [86.77085171670323]
We present a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun.
arXiv Detail & Related papers (2023-06-05T21:26:12Z)
Isotonic Mechanism for Exponential Family Estimation in Machine Learning Peer Review [28.06558596439521]
In 2023, the International Conference on Machine Learning (ICML) required authors with multiple submissions to rank their submissions based on perceived quality.<n>We employ these author-specified rankings to enhance peer review in machine learning and artificial intelligence conferences.<n>We generate adjusted scores that closely align with the original scores while adhering to author-specified rankings.
arXiv Detail & Related papers (2023-04-21T17:59:08Z)
You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism [2.5336146096572745]
We introduce the Isotonic Mechanism to enhance the accuracy of noisy review scores.<n>Authors with multiple submissions are required to rank their papers in descending order of perceived quality.<n> adjusted scores are shown to be more accurate than the raw scores.
arXiv Detail & Related papers (2022-06-14T14:35:53Z)
Ranking Scientific Papers Using Preference Learning [48.78161994501516]
We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
arXiv Detail & Related papers (2021-09-02T19:41:47Z)
State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication [19.10668029301668]
State-Of-the-Art Review (SOAR) is a neoteric reviewing pipeline that serves as a 'plug-and-play' replacement for peer review. At the heart of our approach is an interpretation of the review process as a multi-objective, massively distributed and extremely-high-latency optimisation.
arXiv Detail & Related papers (2020-03-31T17:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.