Evaluating LLM-Based Grant Proposal Review via Structured Perturbations

• URL: http://arxiv.org/abs/2603.08281v2
• Date: Wed, 11 Mar 2026 21:32:58 GMT
• Title: Evaluating LLM-Based Grant Proposal Review via Structured Perturbations
• Abstract summary: We develop a perturbation-based framework probing LLM sensitivity across six quality axes.<n>We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels.
• Score: 18.689211845609623
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap'' for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.

Related papers

• Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards [6.002323709229882]
  Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG)<n>We propose RLAES, a unified LLM framework that jointly optimize essay scoring and feedback generation through RL.<n>To make feedback quality measurable, interpretable, and usable for training, we introduce RL-based Feedback Evaluation (RFE)
  arXiv  Detail & Related papers  (2026-07-21T15:49:02Z)
• BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation [11.37493959290663]
  We introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings.<n>We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges.
  arXiv  Detail & Related papers  (2026-04-10T17:08:40Z)
• Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation [14.465019747508604]
  We propose a new, practical, and efficient approach to LLM failure rate estimation based on constrained maximum-likelihood estimation (MLE)<n>We validate our approach through a comprehensive empirical study, benchmarking it against state-of-the-art baselines like Prediction-Powered Inference (PPI)
  arXiv  Detail & Related papers  (2026-03-11T21:48:40Z)
• Reviewing the Reviewer: Elevating Peer Review Quality through LLM-Guided Feedback [75.31379834079648]
  We introduce an LLM-driven framework that decomposes reviews into argumentative segments.<n>We also release LazyReviewPlus, a dataset of 1,309 sentences labeled for lazy thinking and specificity.
  arXiv  Detail & Related papers  (2026-01-17T20:32:18Z)
• Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning [34.43632129774481]
  In this paper, we quantify and investigate the potential reason -- imbalanced evaluation preference.<n>Motivated by the analysis of the reason, a novel perplexity-aware reinforcement learning algorithm is proposed to rectify the evaluation preference.
  arXiv  Detail & Related papers  (2025-11-13T13:37:45Z)
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation [87.38454788767545]
  We propose IF-CRITIC, an evaluation model for instruction following in Large Language Models.<n>With the scalable reward signals provided by IF-CRITIC, LLMs can achieve substantial performance gains in instruction-following optimization.
  arXiv  Detail & Related papers  (2025-11-02T17:06:49Z)
• AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume [2.2153783542347805]
  This paper investigates challenges in automatic text summarization evaluation.<n>Based on experiments conducted across six representative metrics, we highlight significant discrepancies between reported performances in the literature and those observed in our experimental setting.<n>We introduce a unified, open-source framework, applied to the SummEval dataset and designed to support fair and transparent comparison of evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-29T08:05:00Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• LITE: LLM-Impelled efficient Taxonomy Evaluation [32.8374687916129]
  LITE is a top-down hierarchical taxonomy evaluation strategy, breaking down the taxonomy into manageable substructures.<n>It provides both quantitative performance analysis and qualitative insights by integrating evaluation metrics closely aligned with task objectives.
  arXiv  Detail & Related papers  (2025-04-02T05:33:05Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge [90.8674158031845]
  We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses.<n>This process effectively guides LLM-as-a-Judge to provide a more detailed chain-of-thought (CoT) judgment.<n>Our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling.
  arXiv  Detail & Related papers  (2025-02-18T03:31:06Z)
• Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates [11.948519516797745]
  We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges.<n>Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.
  arXiv  Detail & Related papers  (2024-08-23T11:49:01Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.