Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations

• URL: http://arxiv.org/abs/2604.16318v1
• Date: Mon, 09 Feb 2026 16:44:42 GMT
• Title: Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations
• Abstract summary: Large language models (LLMs) and cross-encoder rerankers have gained attention for improving recommender systems.<n>This paper presents a systematic diagnostic study of cross-encoder rerankers in cold-start movie recommendation using the Serendipity-2018 dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) and cross-encoder rerankers have gained attention for improving recommender systems, particularly in cold-start scenarios where user interaction history is limited. However, practical deployment reveals significant performance gaps between LLM-based approaches and simple baselines. This paper presents a systematic diagnostic study of cross-encoder rerankers in cold-start movie recommendation using the Serendipity-2018 dataset. Through controlled experiments with 500 users across multiple random seeds, we identify three critical failure modes: (1) low retrieval coverage in candidate generation (recall@200 = 0.109 vs. 0.609 for baselines), (2) severe exposure bias with rerankers concentrating recommendations on 3 unique items versus 497 for random baseline, and (3) minimal score discrimination between relevant and irrelevant items (mean difference = 0.098, Cohen's d = 0.13). We demonstrate that popularity-based ranking substantially outperforms LLM reranking (HR@10: 0.268 vs. 0.008, p < 0.001), with the performance gap primarily attributable to retrieval stage limitations rather than reranker capacity. Based on these findings, we provide actionable recommendations including hybrid retrieval strategies, candidate pool size optimization, and score calibration techniques. All code, configurations, and experimental results are made available for reproducibility.

Related papers

• Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation [0.7874708385247353]
  Large language models (LLMs) are increasingly used as rerankers in recommender systems.<n>We compare proprietary, open-weight, and fine-tuned LLM rerankers with collaborative-filtering and sequential baselines.<n>For the strongest proprietary rerankers, switching from semantic to collaborative-filtering candidates raises NDCG@10 by more than 50%.
  arXiv  Detail & Related papers  (2026-08-31T10:54:20Z)
• Optimizing Visual Generative Models via Distribution-wise Rewards [57.46109819595665]
  We present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions.<n>Our approach significantly improves FID-50K across various base models, from 5.77 for SiT and from 3.74 to 3.52 for EDM2.
  arXiv  Detail & Related papers  (2026-07-02T15:08:56Z)
• LLM-Extracted Covariates for Clinical Causal Inference: Rethinking Integration Strategies [7.5675661390024125]
  Causal inference from electronic health records is fundamentally limited by unmeasured confounding.<n>We show how to effectively integrate latent confounders into causal estimation pipelines.
  arXiv  Detail & Related papers  (2026-04-18T00:26:10Z)
• Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE [7.609008983716641]
  We conduct experiments on multimodal sequential recommendation to compare common negative-selection strategies and their interaction with DPO training.<n>Our central finding is that a simple modification, replacing deterministic hard negatives with sampling from a dynamic top-K candidate pool, consistently improves ranking performance.
  arXiv  Detail & Related papers  (2026-03-31T04:49:32Z)
• Greedy Is a Strong Default: Agents as Iterative Optimizers [0.22843885788439797]
  We replace the random proposal generator with an LLM agent that reasons about evaluation diagnostics to propose informed candidates.<n>We evaluate on four tasks spanning discrete, mixed, and continuous search spaces.
  arXiv  Detail & Related papers  (2026-03-28T21:26:40Z)
• Improving LLM-based Recommendation with Self-Hard Negatives from Intermediate Layers [80.55429742713623]
  ILRec is a novel preference fine-tuning framework for LLM-based recommender systems.<n>We introduce a lightweight collaborative filtering model to assign token-level rewards for negative signals.<n>Experiments on three datasets demonstrate ILRec's effectiveness in enhancing the performance of LLM-based recommender systems.
  arXiv  Detail & Related papers  (2026-02-19T14:37:43Z)
• RGAlign-Rec: Ranking-Guided Alignment for Latent Query Reasoning in Recommendation Systems [25.34524038198569]
  We propose RGAlign-Rec, a closed-loop alignment framework for proactive intent prediction.<n>We also introduce Ranking-Guided Alignment (RGA), a multi-stage training paradigm.<n>Our framework achieves a 0.12% gain in GAUC, leading to a significant 3.52% relative reduction in error rate, and a 0.56% improvement in Recall@3.
  arXiv  Detail & Related papers  (2026-02-13T14:38:02Z)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data? [82.09573568241724]
  EssenceBench is a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA)<n>Our approach yields superior compression results with lower reconstruction error and markedly higher efficiency.<n>On the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples.
  arXiv  Detail & Related papers  (2025-10-12T05:38:10Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs [60.58434523646137]
  A popular approach for improving the correctness of output from large language models (LLMs) is Self-Consistency. We introduce Adaptive-Consistency, a cost-efficient, model-agnostic technique that dynamically adjusts the number of samples per question. Our experiments show that Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%.
  arXiv  Detail & Related papers  (2023-05-19T17:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.