Examining the Cognitive Gap Between Authors and Peer Reviewers on Academic Paper Novelty

• URL: http://arxiv.org/abs/2606.13452v1
• Date: Thu, 11 Jun 2026 15:11:24 GMT
• Title: Examining the Cognitive Gap Between Authors and Peer Reviewers on Academic Paper Novelty
• Abstract summary: We analyzed 15,328 academic papers published in Nature Communications from 2016 to 2021, along with their peer-review comments.<n>We found that both reviewers and authors emphasize result-oriented innovation, with reviewers adopting a more comprehensive evaluation perspective.<n>We also found that promotional language significantly correlates with reviewer disagreement on novelty specifically for papers of moderate innovativeness, whereas it has negligible impact for papers with either very high or very low novelty.
• Score: 3.2586114800974957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Novelty is a crucial metric for assessing the quality of academic papers. Scholars strive to highlight the novel aspects of their work, particularly in the title, abstract, and introduction. Peer review, serving as the gatekeeper of scientific rigor, rigorously evaluates the novelty of papers, yet a cognitive gap may exist between author self-promotion and reviewer evaluation. To investigate this, we analyzed 15,328 academic papers published in Nature Communications from 2016 to 2021, along with their peer-review comments. We found that both reviewers and authors emphasize result-oriented innovation, with reviewers adopting a more comprehensive evaluation perspective. Furthermore, by examining promotional intensity against inherent paper novelty, we found that its effect depends on the paper's actual innovation level. Highly innovative papers benefit from stronger promotional language, receiving more positive evaluations. We also found that promotional language significantly correlates with reviewer disagreement on novelty specifically for papers of moderate innovativeness, whereas it has negligible impact for papers with either very high or very low novelty. This reveals how promotional language operates most prominently in the gray area of academic evaluation.

Related papers

• NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment [17.02516373665209]
  We introduce NovBench, the first large-scale benchmark designed to evaluate large language models' capability to generate novelty evaluations.<n> NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations.<n>We propose a four-dimensional evaluation framework to assess the quality of LLM-generated novelty evaluations.
  arXiv  Detail & Related papers  (2026-04-13T14:35:17Z)
• ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review [48.60540055009675]
  ScholarPeer is a search-enabled multi-agent framework designed to emulate the cognitive processes of a senior researcher.<n>We evaluate ScholarPeer on DeepReview-13K and the results demonstrate that ScholarPeer achieves significant win-rates against state-of-the-art approaches in side-by-side evaluations.
  arXiv  Detail & Related papers  (2026-01-30T06:54:55Z)
• Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time [55.756345497678204]
  We introduce a new framework for evidence-based comparative study of review quality.<n>We apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL.<n>We study the relationships between measurements of review quality, and its evolution over time.
  arXiv  Detail & Related papers  (2026-01-21T16:48:29Z)
• Exposía: Academic Writing Assessment of Exposés and Peer Feedback [56.428320613219306]
  We present Exposa, the first public dataset that connects writing and feedback assessment in higher education.<n>We use Exposa to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews.
  arXiv  Detail & Related papers  (2026-01-10T11:33:26Z)
• NoveltyRank: Estimating Conceptual Novelty of AI Papers [8.218640708170119]
  This project aims to develop a model that estimates and ranks conceptual novelty of AI papers.<n>Our approach evaluates novelty primarily through a paper's title, abstract, and semantic similarity to prior literature.<n>We fine-tune Qwen3-4B-Instruct-2507 and SciBERT on both tasks, benchmarking against GPT-5.1 to analyze how task formulation and modeling choices affect performance.
  arXiv  Detail & Related papers  (2025-12-12T03:33:32Z)
• OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models [55.21589313404023]
  OpenReview is a continually evolving repository of research papers, peer reviews, author rebuttals, meta-reviews, and decision outcomes.<n>We highlight three promising areas in which OpenReview can uniquely contribute: enhancing the quality, scalability, and accountability of peer review processes; enabling meaningful, open-ended benchmarks rooted in genuine expert deliberation; and supporting alignment research through real-world interactions reflecting expert assessment, intentions, and scientific values.<n>We suggest the community collaboratively explore standardized benchmarks and usage guidelines around OpenReview, inviting broader dialogue on responsible data use, ethical considerations, and collective stewardship.
  arXiv  Detail & Related papers  (2025-05-24T09:07:13Z)
• In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis [52.42612945266194]
  We propose a new task: generating nuanced, expressive, and time-aware impact summaries.<n>We show that these summaries capture both praise (confirmation citations) and critique (correction citations) through the evolution of fine-grained citation intents.
  arXiv  Detail & Related papers  (2025-05-20T19:11:06Z)
• Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review [4.35783648216893]
  Traditional closed peer review systems are slow, costly, non-transparent, and possibly subject to biases.<n>We propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process.
  arXiv  Detail & Related papers  (2025-01-22T17:00:27Z)
• Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications [12.183473842592567]
  We introduce a scholarly novelty benchmark (SchNovel) to evaluate large language models' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. RAG-Novelty simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty.
  arXiv  Detail & Related papers  (2024-09-25T04:12:38Z)
• From Words to Worth: Newborn Article Impact Prediction with LLM [69.41680520058418]
  This paper introduces a promising approach, leveraging the capabilities of LLMs to predict the future impact of newborn articles.<n>The proposed method employs LLM to discern the shared semantic features of highly impactful papers from a large collection of title-abstract pairs.<n>The quantitative results, with an MAE of 0.216 and an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-08-07T17:52:02Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• A Content-Based Novelty Measure for Scholarly Publications: A Proof of Concept [9.148691357200216]
  We introduce an information-theoretic measure of novelty in scholarly publications. This measure quantifies the degree of'surprise' perceived by a language model that represents the word distribution of scholarly discourse.
  arXiv  Detail & Related papers  (2024-01-08T03:14:24Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.