Optimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning

• URL: http://arxiv.org/abs/2406.14169v1
• Date: Thu, 20 Jun 2024 10:20:02 GMT
• Title: Optimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning
• Authors: Amit Sharma, Hua Li, Xue Li, Jian Jiao,
• Abstract summary: In real-world systems, an important consideration for a new model is novelty of its top-k recommendations. We propose a reinforcement learning (RL) formulation where large language models provide feedback for the novel items. We evaluate the proposed algorithm on improving novelty for a query-ad recommendation task on a large-scale search engine.
• Score: 16.287067991245962
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given an input query, a recommendation model is trained using user feedback data (e.g., click data) to output a ranked list of items. In real-world systems, besides accuracy, an important consideration for a new model is novelty of its top-k recommendations w.r.t. an existing deployed model. However, novelty of top-k items is a difficult goal to optimize a model for, since it involves a non-differentiable sorting operation on the model's predictions. Moreover, novel items, by definition, do not have any user feedback data. Given the semantic capabilities of large language models, we address these problems using a reinforcement learning (RL) formulation where large language models provide feedback for the novel items. However, given millions of candidate items, the sample complexity of a standard RL algorithm can be prohibitively high. To reduce sample complexity, we reduce the top-k list reward to a set of item-wise rewards and reformulate the state space to consist of <query, item> tuples such that the action space is reduced to a binary decision; and show that this reformulation results in a significantly lower complexity when the number of items is large. We evaluate the proposed algorithm on improving novelty for a query-ad recommendation task on a large-scale search engine. Compared to supervised finetuning on recent <query, ad> pairs, the proposed RL-based algorithm leads to significant novelty gains with minimal loss in recall. We obtain similar results on the ORCAS query-webpage matching dataset and a product recommendation dataset based on Amazon reviews.

Related papers

• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• Ordinal Preference Optimization: Aligning Human Preferences via NDCG [28.745322441961438]
  We develop an end-to-end preference optimization algorithm by approxing NDCG with a differentiable surrogate loss. OPO outperforms existing pairwise and listwise approaches on evaluation sets and general benchmarks like AlpacaEval.
  arXiv  Detail & Related papers  (2024-10-06T03:49:28Z)
• Large Language Models for Relevance Judgment in Product Search [48.56992980315751]
  High relevance of retrieved and re-ranked items to the search query is the cornerstone of successful product search. We present an array of techniques for leveraging Large Language Models (LLMs) for automating the relevance judgment of query-item pairs (QIPs) at scale. Our findings have immediate implications for the growing field of relevance judgment automation in product search.
  arXiv  Detail & Related papers  (2024-06-01T00:52:41Z)
• Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversational Search [25.16282868262589]
  RetPO is designed to optimize a language model (LM) for reformulating search queries in line with the preferences of the target retrieval systems. We construct a large-scale dataset called Retrievers' Feedback on over 410K query rewrites across 12K conversations. The resulting model achieves state-of-the-art performance on two recent conversational search benchmarks.
  arXiv  Detail & Related papers  (2024-02-19T04:41:31Z)
• List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation [80.12531449946655]
  We propose a Reranking-Truncation joint model (GenRT) that can perform the two tasks concurrently. GenRT integrates reranking and truncation via generative paradigm based on encoder-decoder architecture. Our method achieves SOTA performance on both reranking and truncation tasks for web search and retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2024-02-05T06:52:53Z)
• Revisiting Neural Retrieval on Accelerators [20.415728886298915]
  A key component of retrieval is to model (user, item) similarity. Despite its popularity, dot products cannot capture complex user-item interactions, which are multifaceted and likely high rank. We propose textitmixture of logits (MoL), which models (user, item) similarity as an adaptive composition of elementary similarity functions.
  arXiv  Detail & Related papers  (2023-06-06T22:08:42Z)
• Zero-Shot Listwise Document Reranking with a Large Language Model [58.64141622176841]
  We propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker.
  arXiv  Detail & Related papers  (2023-05-03T14:45:34Z)
• Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models [6.425088990363101]
  We examine the relationship between fluency and attribution in Large Language Models prompted with retrieved evidence. We show that larger models tend to do much better in both fluency and attribution. We propose a recipe that could allow smaller models to both close the gap with larger models and preserve the benefits of top-k retrieval.
  arXiv  Detail & Related papers  (2023-02-11T02:43:34Z)
• Fine-grained Retrieval Prompt Tuning [149.9071858259279]
  Fine-grained Retrieval Prompt Tuning steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompt and feature adaptation. Our FRPT with fewer learnable parameters achieves the state-of-the-art performance on three widely-used fine-grained datasets.
  arXiv  Detail & Related papers  (2022-07-29T04:10:04Z)
• Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling [58.14878401145309]
  We develop a novel approach to producing more sample-efficient estimators of expectations in the PL model. We illustrate our findings both theoretically and empirically using real-world recommendation data from Amazon Music and the Yahoo learning-to-rank challenge.
  arXiv  Detail & Related papers  (2022-05-12T11:15:47Z)
• Top-N Recommendation with Counterfactual User Preference Simulation [26.597102553608348]
  Top-N recommendation, which aims to learn user ranking-based preference, has long been a fundamental problem in a wide range of applications. In this paper, we propose to reformulate the recommendation task within the causal inference framework to handle the data scarce problem.
  arXiv  Detail & Related papers  (2021-09-02T14:28:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.