Do LLM-judges Align with Human Relevance in Cranfield-style Recommender Evaluation?

• URL: http://arxiv.org/abs/2511.23312v1
• Date: Fri, 28 Nov 2025 16:10:39 GMT
• Title: Do LLM-judges Align with Human Relevance in Cranfield-style Recommender Evaluation?
• Abstract summary: This paper investigates whether Large Language Models (LLMs) can serve as reliable automatic judges to address scalability challenges.<n>Using the ML-32M-ext Cranfield-style movie recommendation collection, we first examine the limitations of existing evaluation methodologies.<n>We find that incorporating richer item metadata and longer user histories improves alignment, and that LLM-judge yields high agreement with human-based rankings.
• Score: 40.49875426230813
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Evaluating recommender systems remains a long-standing challenge, as offline methods based on historical user interactions and train-test splits often yield unstable and inconsistent results due to exposure bias, popularity bias, sampled evaluations, and missing-not-at-random patterns. In contrast, textual document retrieval benefits from robust, standardized evaluation via Cranfield-style test collections, which combine pooled relevance judgments with controlled setups. While recent work shows that adapting this methodology to recommender systems is feasible, constructing such collections remains costly due to the need for manual relevance judgments, thus limiting scalability. This paper investigates whether Large Language Models (LLMs) can serve as reliable automatic judges to address these scalability challenges. Using the ML-32M-ext Cranfield-style movie recommendation collection, we first examine the limitations of existing evaluation methodologies. Then we explore the alignment and the recommender systems ranking agreement between the LLM-judge and human provided relevance labels. We find that incorporating richer item metadata and longer user histories improves alignment, and that LLM-judge yields high agreement with human-based rankings (Kendall's tau = 0.87). Finally, an industrial case study in the podcast recommendation domain demonstrates the practical value of LLM-judge for model selection. Overall, our results show that LLM-judge is a viable and scalable approach for evaluating recommender systems.

Related papers

• User Preference Induction with LLMs for Offline Top-N Recommendation Evaluation [5.10348690267577]
  We propose a framework to expand relevance judgements for offline recommender evaluation.<n>Our approach uses large language models in two complementary roles.<n> Experimental results show that this approach is a promising strategy for improving the robustness of offline top-N evaluation.
  arXiv  Detail & Related papers  (2026-07-13T10:16:24Z)
• Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs [34.14678608130442]
  The unjudged document problem is a key obstacle to the reusability of test collections in information retrieval.<n>We train topic-specific relevance classifiers by finetuning monoT5 with independent LoRA weight adaptation on the judgments of a single assessor.<n>As little as 128 initial human judgments per topic suffice to improve the comparability of models.
  arXiv  Detail & Related papers  (2025-10-06T09:38:13Z)
• RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback [14.456969388690066]
  RankArena is a unified platform for comparing and analysing the performance of retrieval pipelines, rerankers, and RAG systems.<n>It supports multiple evaluation modes: direct reranking visualisation, blind pairwise comparisons with human or LLM voting, supervised manual document annotation, and end-to-end RAG answer quality assessment.<n>It captures fine-grained relevance feedback through both pairwise preferences and full-list annotations, along with auxiliary metadata such as movement metrics, annotation time, and quality ratings.
  arXiv  Detail & Related papers  (2025-08-07T15:46:53Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• Quantitative LLM Judges [60.773734899532336]
  We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain.<n>The models are trained to improve the score of the original judge using its rationale and score.<n>Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.
  arXiv  Detail & Related papers  (2025-06-03T14:44:23Z)
• Reverse Engineering Human Preferences with Reinforcement Learning [15.70441204657966]
  Large Language Models (LLMs) are routinely evaluated by other LLMs trained to predict human preferences.<n>Previous work shows that the answers generated by a candidate-LLM can be edited post hoc to maximise the score assigned to them by a judge-LLM.<n>We adopt a different approach and use the signal provided by judge-LLMs as a reward to adversarially tune models.
  arXiv  Detail & Related papers  (2025-05-21T17:48:16Z)
• Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models [68.92020689188887]
  Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs)<n>Existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation.<n>This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models.
  arXiv  Detail & Related papers  (2025-02-26T04:50:43Z)
• JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment [28.4353755578306]
  Large Language Models (LLMs) have shown promise in generating relevance labels for search tasks.<n>We introduce JudgeBlender, a framework that employs smaller, open-source models to provide relevance judgments.
  arXiv  Detail & Related papers  (2024-12-17T19:04:15Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System [9.470545149911072]
  This paper proposes a normative framework to benchmark consumer fairness in LLM-powered recommender systems. We argue that this gap can lead to arbitrary conclusions about fairness. Experiments on the MovieLens dataset on consumer fairness reveal fairness deviations in age-based recommendations.
  arXiv  Detail & Related papers  (2024-05-03T16:25:27Z)
• Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation [52.62492168507781]
  We propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM) This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations.
  arXiv  Detail & Related papers  (2023-05-12T16:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.