Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations

• URL: http://arxiv.org/abs/2609.16304v1
• Date: Mon, 14 Sep 2026 20:09:12 GMT
• Title: Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations
• Abstract summary: Large language models (LLMs) are increasingly used for product recommendation.<n> evaluating their recommendations presents challenges that differ from conventional information retrieval and recommender systems.<n>We introduce a framework for evaluating open-ended LLM brand recommendations.
• Score: 2.7506573311649762
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used for product recommendation, but evaluating their recommendations presents challenges that differ from conventional information retrieval and recommender systems. LLMs can generate recommendations without an explicit candidate set, and repeated responses to the same query can produce different brands and rankings. We introduce a framework for evaluating open-ended LLM brand recommendations that defines the competitive set independently of model outputs and estimates recommendation prevalence and prominence through repeated sampling. We operationalize these constructs using Brand Recommendation Probability (BRP@$k$) and Mean Reciprocal Rank (MRR@$k$), and apply the framework to six LLMs across five product categories. Category-only queries reveal substantial omission of established brands and limited evidence that recommendation prominence follows conventional brand popularity. Instead, prominence is associated with broader marketplace-visibility signals, particularly search interest and online brand conversation. Needs-based queries show that contextualizing users' goals and constraints changes which brands are retrieved, while diagnostic positioning probes demonstrate that brands omitted from ordinary recommendations can remain conditionally retrievable when distinctive cues are supplied. These findings highlight the need to evaluate LLM recommendation as a stochastic retrieval-and-ranking process rather than from individual generated lists. We provide open-source software and data to support reproducible evaluation of LLM-generated brand recommendations.

Related papers

• Evaluating LLM-Based Mobile App Recommendations: An Empirical Study [1.9932595193134042]
  Large Language Models (LLMs) are increasingly used to recommend mobile applications through natural language prompts.<n>We present an empirical analysis of how widely-used general purpose LLMs generate, justify, and rank mobile app recommendations.
  arXiv  Detail & Related papers  (2025-10-21T07:35:19Z)
• CARE: Contextual Adaptation of Recommenders for LLM-based Conversational Recommendation [66.51329063956538]
  We introduce the CARE (Contextual Adaptation of Recommenders) framework.<n> CARE customizes large language models for CRS tasks, and synergizes them with external recommendation systems.<n>Our results demonstrate that incorporating external recommender systems with entity-level information significantly enhances recommendation accuracy of CRS.
  arXiv  Detail & Related papers  (2025-08-19T14:53:30Z)
• Towards Comprehensible Recommendation with Large Language Model Fine-tuning [41.218487308635126]
  We propose a novel Content Understanding from a Collaborative Perspective framework (CURec) for recommendation systems.<n>Curec generates collaborative-aligned content features for more comprehensive recommendations.<n>Experiments on public benchmarks demonstrate the superiority of CURec over existing methods.
  arXiv  Detail & Related papers  (2025-08-11T03:55:31Z)
• Generative Product Recommendations for Implicit Superlative Queries [21.750990820244983]
  In Recommender Systems, users often seek the best products through indirect, vague, or under-specified queries, such as "best shoes for trail running"<n>We investigate how Large Language Models can generate implicit attributes for ranking as well as reason over them to improve product recommendations for such queries.
  arXiv  Detail & Related papers  (2025-04-26T00:05:47Z)
• Enhancing Recommendation Diversity by Re-ranking with Large Language Models [0.27624021966289597]
  Large Language Models (LLMs) can be used for diversity re-ranking. LLMs exhibit improved performance on many natural language processing and recommendation tasks.
  arXiv  Detail & Related papers  (2024-01-21T14:33:52Z)
• LLMRec: Benchmarking Large Language Models on Recommendation Task [54.48899723591296]
  The application of Large Language Models (LLMs) in the recommendation domain has not been thoroughly investigated. We benchmark several popular off-the-shelf LLMs on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation.
  arXiv  Detail & Related papers  (2023-08-23T16:32:54Z)
• A Survey on Large Language Models for Recommendation [77.91673633328148]
  Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) This survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec)
  arXiv  Detail & Related papers  (2023-05-31T13:51:26Z)
• Large Language Models are Zero-Shot Rankers for Recommender Systems [76.02500186203929]
  This work aims to investigate the capacity of large language models (LLMs) to act as the ranking model for recommender systems. We show that LLMs have promising zero-shot ranking abilities but struggle to perceive the order of historical interactions. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies.
  arXiv  Detail & Related papers  (2023-05-15T17:57:39Z)
• Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation [52.62492168507781]
  We propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM) This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations.
  arXiv  Detail & Related papers  (2023-05-12T16:54:36Z)
• Recommendation Systems with Distribution-Free Reliability Guarantees [83.80644194980042]
  We show how to return a set of items rigorously guaranteed to contain mostly good items. Our procedure endows any ranking model with rigorous finite-sample control of the false discovery rate. We evaluate our methods on the Yahoo! Learning to Rank and MSMarco datasets.
  arXiv  Detail & Related papers  (2022-07-04T17:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.