LLMs Can Assist with Proposal Selection at Large User Facilities

• URL: http://arxiv.org/abs/2512.10895v1
• Date: Thu, 11 Dec 2025 18:23:56 GMT
• Title: LLMs Can Assist with Proposal Selection at Large User Facilities
• Abstract summary: Large language models (LLMs) can enhance the proposal selection process at large user facilities.<n>LLMs provide a scalable, consistent, and cost-effective alternative to traditional human review.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore how large language models (LLMs) can enhance the proposal selection process at large user facilities, offering a scalable, consistent, and cost-effective alternative to traditional human review. Proposal selection depends on assessing the relative strength among submitted proposals; however, traditional human scoring often suffers from weak inter-proposal correlations and is subject to reviewer bias and inconsistency. A pairwise preference-based approach is logically superior, providing a more rigorous and internally consistent basis for ranking, but its quadratic workload makes it impractical for human reviewers. We address this limitation using LLMs. Leveraging the uniquely well-curated proposals and publication records from three beamlines at the Spallation Neutron Source (SNS), Oak Ridge National Laboratory (ORNL), we show that the LLM rankings correlate strongly with the human rankings (Spearman $ρ\simeq 0.2-0.8$, improving to $\geq 0.5$ after 10\% outlier removal). Moreover, LLM performance is no worse than that of human reviewers in identifying proposals with high publication potential, while costing over two orders of magnitude less. Beyond ranking, LLMs enable advanced analyses that are challenging for humans, such as quantitative assessment of proposal similarity via embedding models, which provides information crucial for review committees.

Related papers

• Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need? [1.111977509278551]
  Large language models (LLMs) are increasingly used as automated evaluators of AI systems, including in high-stakes applications.<n> Expert human ratings are costly and difficult to scale, whereas LLM ratings can be produced quickly at low cost.<n>This paper shifts the role of the LLM judge from substitutive to auxiliary, and formulates the LLM-as-a-judge paradigm as one of augmenting human evaluation.
  arXiv  Detail & Related papers  (2026-05-08T17:13:08Z)
• Towards Position-Robust Talent Recommendation via Large Language Models [2.145037859243627]
  We propose L3TR, a novel framework for listwise talent recommendation with large language models (LLMs)<n>In this paper, we propose a block attention mechanism and a local positional encoding method to enhance inter-document processing and mitigate the position bias and concurrent token bias issue.<n>Experiments on two real-world datasets validated the effectiveness of L3TR, showing consistent improvements over existing baselines.
  arXiv  Detail & Related papers  (2026-04-02T15:54:03Z)
• Automated Concept Discovery for LLM-as-a-Judge Preference Analysis [21.171990974350773]
  Large Language Models (LLMs) are increasingly used as scalable evaluators of model outputs.<n>Their preference judgments exhibit systematic biases and can diverge from human evaluations.<n>We study several embedding-level concept extraction methods for analyzing LLM judge behavior.
  arXiv  Detail & Related papers  (2026-02-09T20:55:16Z)
• Evaluating Position Bias in Large Language Model Recommendations [3.430780143519032]
  Large Language Models (LLMs) are being increasingly explored as general-purpose tools for recommendation tasks.<n>We show that LLM-based recommendation models suffer from position bias, where the order of candidate items in a prompt can disproportionately influence the recommendations produced by LLMs.<n>We introduce a new prompting strategy to mitigate the position bias of LLM recommendation models called Ranking via Iterative SElection.
  arXiv  Detail & Related papers  (2025-08-04T03:30:26Z)
• RocketEval: Efficient Automated LLM Evaluation via Grading Checklist [32.66840523942929]
  We propose a straightforward, replicable, and accurate automated evaluation method by leveraging a lightweight LLM as the judge, named RocketEval.<n>Our experiments carried out on the automated evaluation benchmarks, MT-Bench and WildBench datasets, reveal that RocketEval, when using Gemma-2-2B as the judge, achieves a high correlation (0.965) with human preferences.
  arXiv  Detail & Related papers  (2025-03-07T04:51:30Z)
• Tuning LLM Judge Design Decisions for 1/1000 of the Cost [42.06346155380305]
  Large Language Models (LLMs) often require costly human annotations.<n>To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs.<n>While several approaches have been proposed, many confounding factors are present between different papers.
  arXiv  Detail & Related papers  (2025-01-24T17:01:14Z)
• LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? [20.998805709422292]
  Test collections are information-retrieval tools that allow researchers to quickly and easily evaluate ranking algorithms.<n>As a cheaper alternative, recent studies have proposed using large language models (LLMs) to completely replace human assessors.<n>We propose LARA, an effective method to balance manual annotations with LLM annotations, helping build a rich and reliable test collection even under a low budget.
  arXiv  Detail & Related papers  (2024-11-11T11:17:35Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments [41.25558612970942]
  We show that large language models (LLMs) exhibit preference biases and worrying sensitivity to prompt designs. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO.
  arXiv  Detail & Related papers  (2024-06-17T09:48:53Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)
• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models [55.60306377044225]
  Large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks. This paper explores two options for exploiting the emergent abilities of LLMs for zero-shot NLG assessment. For moderate-sized open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is superior to prompt scoring.
  arXiv  Detail & Related papers  (2023-07-15T22:02:12Z)
• Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization [66.08074487429477]
  We investigate the stability and reliability of large language models (LLMs) as automatic evaluators for abstractive summarization. We find that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements.
  arXiv  Detail & Related papers  (2023-05-22T14:58:13Z)
• Benchmarking Large Language Models for News Summarization [79.37850439866938]
  Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. We find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability.
  arXiv  Detail & Related papers  (2023-01-31T18:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.