On the Convergent Validity of Offline Evaluation Designs for Recommender Systems

• URL: http://arxiv.org/abs/2607.25097v1
• Date: Mon, 27 Jul 2026 21:54:19 GMT
• Title: On the Convergent Validity of Offline Evaluation Designs for Recommender Systems
• Abstract summary: We study how offline evaluation design choices affect the validity of recommender system comparisons.<n>We measure the correlation between model rankings obtained from conventional train-test splits on sparse interaction data and rankings from evaluations based on dense ground-truth user feedback.<n>Our results show that the validity of sparse evaluation depends on the dataset and the specific dense evaluation targets, and that there is no uniformly best offline evaluation design.
• Score: 1.7682678511634797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline evaluation on historical interaction logs is the most common evaluation methodology for recommender systems. However, such evaluations depend on sparse, incomplete, or biased data, which raises concerns about whether commonly used evaluation setups reliably reflect true user preferences. In this work, we study how offline evaluation design choices affect the validity of recommender system comparisons. We evaluate a set of recommendation models across several evaluation setups that vary key factors such as data filtering thresholds and candidate set construction. To assess the validity of these configurations, we measure the correlation between model rankings obtained from conventional train-test splits on sparse interaction data and rankings from evaluations based on dense ground-truth user feedback. We use this agreement as an indication of their validity with respect to true user preferences. Our results show that the validity of sparse evaluation depends on the dataset and the specific dense evaluation targets, and that there is no uniformly best offline evaluation design.

Related papers

• User Preference Induction with LLMs for Offline Top-N Recommendation Evaluation [5.10348690267577]
  We propose a framework to expand relevance judgements for offline recommender evaluation.<n>Our approach uses large language models in two complementary roles.<n> Experimental results show that this approach is a promising strategy for improving the robustness of offline top-N evaluation.
  arXiv  Detail & Related papers  (2026-07-13T10:16:24Z)
• LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation [16.60711435187638]
  offline evaluation treats observed user feedback as a proxy of true preferences.<n>Top-K metrics only establish numerical scores without offering meaningful insights to support them.<n>We propose a reliable and explainable LLM-as-a-Judge framework for offline recommendation evaluation.
  arXiv  Detail & Related papers  (2026-06-22T07:42:01Z)
• Offline Preference-Based Trajectory Evaluation [54.93316418589433]
  We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles.<n>We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%.
  arXiv  Detail & Related papers  (2026-06-16T05:42:19Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging [95.02210956333374]
  We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
  arXiv  Detail & Related papers  (2026-05-19T10:59:19Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• On the Reliability of Sampling Strategies in Offline Recommender Evaluation [3.4956406636452626]
  offline evaluation plays a central role in benchmarking recommender systems when online testing is impractical or risky.<n>It is susceptible to two key sources of bias: exposure bias, where users only interact with items they are shown, and sampling bias, introduced when evaluation is performed on a subset of logged items rather than the full catalog.
  arXiv  Detail & Related papers  (2025-08-07T13:50:05Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems [6.540293515339111]
  offline evaluation of retrieval-ranking systems is crucial for developing high-performing models.<n>We propose a novel framework for robust offline evaluation of retrieval-ranking systems.<n>Our contributions include (1) a causal formulation for addressing offline evaluation biases, (2) a system-agnostic debiasing framework, and (3) empirical validation of its effectiveness.
  arXiv  Detail & Related papers  (2025-04-04T23:52:57Z)
• Evaluating the Robustness of Off-Policy Evaluation [10.760026478889664]
  Off-policy Evaluation (OPE) evaluates the performance of hypothetical policies leveraging only offline log data. It is particularly useful in applications where the online interaction involves high stakes and expensive setting. We develop Interpretable Evaluation for Offline Evaluation (IEOE), an experimental procedure to evaluate OPE estimators' robustness.
  arXiv  Detail & Related papers  (2021-08-31T09:33:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.