Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation

• URL: http://arxiv.org/abs/2609.00086v1
• Date: Mon, 31 Aug 2026 10:54:20 GMT
• Title: Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
• Abstract summary: Large language models (LLMs) are increasingly used as rerankers in recommender systems.<n>We compare proprietary, open-weight, and fine-tuned LLM rerankers with collaborative-filtering and sequential baselines.<n>For the strongest proprietary rerankers, switching from semantic to collaborative-filtering candidates raises NDCG@10 by more than 50%.
• Score: 0.7874708385247353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used as rerankers in conversational recommender systems, yet measured gains depend strongly on the retrieval and inference protocol. On the ReDial conversational movie recommendation benchmark, we compare proprietary, open-weight, and fine-tuned LLM rerankers with collaborative-filtering and sequential baselines in a shared retrieve-then-rerank pipeline. We vary candidate-pool size, first-stage retriever, and decoding temperature. With a shared semantic top-250 candidate pool and strict candidate-aware scoring, the best proprietary reranker reaches NDCG@10 of 0.1497, compared with 0.0939 for the strongest non-LLM baseline. The same reranker reaches 0.2925 in zero-shot generation, showing that unconstrained scoring can yield a much larger apparent advantage than matched-pool evaluation. No evaluated open-weight LLM outperforms the tuned shallow autoencoder baseline under this protocol. For the strongest proprietary and open-weight rerankers, switching from semantic to collaborative-filtering candidates raises NDCG@10 by more than 50%, showing that measured reranker performance is highly sensitive to candidate generation. For the best proprietary reranker, raising temperature from 0 to 1.0 increases top-10 Jaccard distance from 0.0900 to 0.1240 while mean NDCG@10 changes negligibly, whereas weaker LLMs show larger degradation. These ReDial results support treating candidate generation, candidate-pool size, scoring policy, and decoding configuration as required reporting fields rather than implementation details.

Related papers

• Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design [16.443448156662644]
  Modern de novo design generate many candidate protein binders, but wet-lab validation capacity remains limited.<n>We study whether LLMs can generate multi-metric ranking policies from precomputed structural-confidence and interface-quality proxy scores.
  arXiv  Detail & Related papers  (2026-08-21T05:32:26Z)
• Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration [8.515185888009833]
  Test-time collaboration is often credited with improving LLM reasoning, yet its gains are uneven and sometimes negative.<n>For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors.<n>This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology.
  arXiv  Detail & Related papers  (2026-07-20T04:09:19Z)
• Improving LLM-Based Go Code Review through Issue-List Generation and Context Augmentation [20.19657859180513]
  We propose an issue-list review paradigm, in which LLMs enumerate all potential issues rather than reporting only the single most important one.<n>We then compare three types of code context augmentation -- neighboring, LSP-based semantics, and IR-based similar co-change context.<n>We integrate candidates from no-context and context-enhanced generation to improve review coverage, and introduce refinement-guided pruning to keep the candidate list at a practical size.
  arXiv  Detail & Related papers  (2026-06-01T08:11:34Z)
• OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation [53.88666485159289]
  We introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley-Terry comparison.<n>OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds.<n>We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.
  arXiv  Detail & Related papers  (2026-05-14T17:57:40Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• Mira-Embeddings-V1: Domain-Adapted Semantic Reranking for Recruitment via LLM-Synthesized Data [12.621394200451613]
  We present mira-embeddings-v1, a semantic reranking system for the recruitment domain.<n>Starting from real JDs, we build a five-stage prompt pipeline to generate diverse positive and hard negative samples.<n>We then apply a two-round LoRA adaptation: JD--JD contrastive training followed by JD--CV triplet alignment on a heterogeneous text dataset.
  arXiv  Detail & Related papers  (2026-04-20T02:51:12Z)
• Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations [0.0]
  Large language models (LLMs) and cross-encoder rerankers have gained attention for improving recommender systems.<n>This paper presents a systematic diagnostic study of cross-encoder rerankers in cold-start movie recommendation using the Serendipity-2018 dataset.
  arXiv  Detail & Related papers  (2026-02-09T16:44:42Z)
• Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning [70.6126069527741]
  ConvRec-R1 is a two-stage framework for end-to-end training of conversational recommender systems.<n>In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline.<n>In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization.
  arXiv  Detail & Related papers  (2025-10-23T02:56:00Z)
• AIME: AI System Optimization via Multiple LLM Evaluators [79.03422337674664]
  AIME is an evaluation protocol that utilizes multiple LLMs that each independently generate an evaluation on separate criteria and then combine them via concatenation. We show AIME outperforming baseline methods in code generation tasks, with up to $62%$ higher error detection rate and up to $16%$ higher success rate than a single LLM evaluation protocol on LeetCodeHard and HumanEval datasets.
  arXiv  Detail & Related papers  (2024-10-04T04:03:24Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.