GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing

• URL: http://arxiv.org/abs/2605.27204v1
• Date: Tue, 26 May 2026 15:58:49 GMT
• Title: GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing
• Abstract summary: We propose a graph-based LLM framework that formulates paper evaluation as review-signal message passing over a semantic paper graph.<n>The graph jointly captures intrinsic quality, synchronic links among contemporaneous papers, and diachronic links to prior work.<n>Experiments show that GraphReview consistently outperforms the strongest baseline, achieving average improvements of 29.7% on decision and ranking metrics.
• Score: 4.802002747361241
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific paper evaluation often involves not only assessing a manuscript itself, but also relating it to contemporaneous research and prior literature. However, existing LLM-based methods typically model these signals separately and lack a unified mechanism for propagating review evidence across papers. We propose $\textbf{GraphReview}$, a graph-based LLM framework that formulates paper evaluation as review-signal message passing over a semantic paper graph. The graph jointly captures intrinsic quality, synchronic links among contemporaneous papers, and diachronic links to prior work. LLMs are used to estimate node-level quality priors and generate edge-level comparative evidence through pairwise paper comparisons, while Personalized PageRank integrates review signals for quality ranking, decision prediction, and review generation. To produce higher-quality graph evidence, we propose reward-induced maximum likelihood objectives for training the LLM backbones. Experiments show that GraphReview consistently outperforms the strongest baseline, achieving average improvements of 29.7% on decision and ranking metrics, including gains of 23.7% in Accuracy and 57.6% in Spearman's $ρ$. It also produces higher-quality review texts and generalizes effectively across time periods and conference venues. The code is available at https://github.com/ECNU-Text-Computing/GraphReview.

Related papers

• SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence [7.2640055022009635]
  We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence.<n>Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set.
  arXiv  Detail & Related papers  (2026-07-29T15:54:14Z)
• Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction [38.170020271877085]
  We present the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction.<n>We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models.
  arXiv  Detail & Related papers  (2026-07-14T14:52:42Z)
• Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators [2.952559770088323]
  Large language models (LLMs) offer new opportunities for automated research evaluation based on textual content.<n>This study examines whether LLMs can support post-publication peer review tasks by benchmarking their outputs against expert judgments and citation-based indicators.
  arXiv  Detail & Related papers  (2026-03-27T11:32:18Z)
• CRISP: Characterizing Relative Impact of Scholarly Publications [70.25915589244961]
  We propose CRISP, which jointly ranks all cited papers within a citing paper using large language models (LLMs)<n>To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting.<n>CRISP outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations.
  arXiv  Detail & Related papers  (2026-03-25T16:42:30Z)
• From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation [12.13840753234467]
  Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently.<n>We propose shifting paper evaluation from isolated scoring to collaborative ranking.<n>Our framework achieves an average relative improvement of textbf21.8% over the strong baseline DeepReview-14B.
  arXiv  Detail & Related papers  (2026-03-18T10:55:02Z)
• ReviewGraph: A Knowledge Graph Embedding Based Framework for Review Rating Prediction with Sentiment Features [3.195234044113248]
  This work proposes ReviewGraph for Review Rating Prediction (RRP), a novel framework that transforms textual customer reviews into knowledge graphs.<n>Using graph embeddings (Node2Vec) and sentiment features, the framework predicts review rating scores through machine learning classifiers.<n>We will share ReviewGraph output and platform open-sourced on our GitHub page.
  arXiv  Detail & Related papers  (2025-08-19T15:44:27Z)
• Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models [70.03602551880526]
  We introduce ProGraph, a benchmark for large language models (LLMs) to process graphs.<n>Our findings reveal that the performance of current LLMs is unsatisfactory, with the best model achieving only 36% accuracy.<n>We propose LLM4Graph datasets, which include crawled documents and auto-generated codes based on 6 widely used graph libraries.
  arXiv  Detail & Related papers  (2024-09-29T11:38:45Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation [87.44350003888646]
  Eval-Instruct can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting. CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines.
  arXiv  Detail & Related papers  (2023-11-30T16:52:42Z)
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment [64.01972723692587]
  We present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin.
  arXiv  Detail & Related papers  (2023-03-29T12:46:54Z)
• FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations [114.94628499698096]
  We propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MRs) MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%.
  arXiv  Detail & Related papers  (2022-04-13T16:45:33Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.