A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels

• URL: http://arxiv.org/abs/2105.06275v1
• Date: Thu, 13 May 2021 13:14:59 GMT
• Title: A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels
• Authors: Nicol\`o Felicioni, Maurizio Ferrari Dacrema, Paolo Cremonesi
• Abstract summary: Video-on-demand and music streaming services provide the user with a page consisting of several recommendation lists. Finding efficient strategies to select which carousels to display is an active research topic of great industrial interest. We propose an offline evaluation protocol for a carousel setting in which the recommendation quality of a model is measured by how much it improves upon that of an already available set of carousels.
• Score: 7.8851236034886645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many video-on-demand and music streaming services provide the user with a page consisting of several recommendation lists, i.e. widgets or swipeable carousels, each built with a specific criterion (e.g. most recent, TV series, etc.). Finding efficient strategies to select which carousels to display is an active research topic of great industrial interest. In this setting, the overall quality of the recommendations of a new algorithm cannot be assessed by measuring solely its individual recommendation quality. Rather, it should be evaluated in a context where other recommendation lists are already available, to account for how they complement each other. This is not considered by traditional offline evaluation protocols. Hence, we propose an offline evaluation protocol for a carousel setting in which the recommendation quality of a model is measured by how much it improves upon that of an already available set of carousels. We report experiments on publicly available datasets on the movie domain and notice that under a carousel setting the ranking of the algorithms change. In particular, when a SLIM carousel is available, matrix factorization models tend to be preferred, while item-based models are penalized. We also propose to extend ranking metrics to the two-dimensional carousel layout in order to account for a known position bias, i.e. users will not explore the lists sequentially, but rather concentrate on the top-left corner of the screen.

Related papers

• Improving Recommendation Relevance by simulating User Interest [77.34726150561087]
  We observe that recommendation "recency" can be straightforwardly and transparently maintained by iterative reduction of ranks of inactive items. The basic idea behind this work is patented in a context of online recommendation systems.
  arXiv  Detail & Related papers  (2023-02-03T03:35:28Z)
• Recommendation Systems with Distribution-Free Reliability Guarantees [83.80644194980042]
  We show how to return a set of items rigorously guaranteed to contain mostly good items. Our procedure endows any ranking model with rigorous finite-sample control of the false discovery rate. We evaluate our methods on the Yahoo! Learning to Rank and MSMarco datasets.
  arXiv  Detail & Related papers  (2022-07-04T17:49:25Z)
• Introducing a Framework and a Decision Protocol to Calibrate Recommender Systems [0.0]
  This paper proposes an approach to create recommendation lists with a calibrated balance of genres. The main claim is that calibration can contribute positively to generate fairer recommendations. We propose a conceptual framework and a decision protocol to generate more than one thousand combinations of calibrated systems.
  arXiv  Detail & Related papers  (2022-04-07T19:30:55Z)
• Utilizing Textual Reviews in Latent Factor Models for Recommender Systems [1.7361353199214251]
  We propose a recommender algorithm that combines a rating modelling technique with a topic modelling method based on textual reviews. We evaluate the performance of the algorithm using Amazon.com datasets with different sizes, corresponding to 23 product categories.
  arXiv  Detail & Related papers  (2021-11-16T15:07:51Z)
• Set2setRank: Collaborative Set to Set Ranking for Implicit Feedback based Recommendation [59.183016033308014]
  In this paper, we explore the unique characteristics of the implicit feedback and propose Set2setRank framework for recommendation. Our proposed framework is model-agnostic and can be easily applied to most recommendation prediction approaches.
  arXiv  Detail & Related papers  (2021-05-16T08:06:22Z)
• Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels [7.8851236034886645]
  It is common for video-on-demand and music streaming services to adopt a user interface composed of several recommendation lists. We propose a two-dimensional evaluation protocol for a carousel setting that will measure the quality of a recommendation carousel.
  arXiv  Detail & Related papers  (2021-05-14T20:33:51Z)
• Dynamic-K Recommendation with Personalized Decision Boundary [41.70842736417849]
  We develop a dynamic-K recommendation task as a joint learning problem with both ranking and classification objectives. We extend two state-of-the-art ranking-based recommendation methods, i.e., BPRMF and HRM, to the corresponding dynamic-K versions. Our experimental results on two datasets show that the dynamic-K models are more effective than the original fixed-N recommendation methods.
  arXiv  Detail & Related papers  (2020-12-25T13:02:57Z)
• CRACT: Cascaded Regression-Align-Classification for Robust Visual Tracking [97.84109669027225]
  We introduce an improved proposal refinement module, Cascaded Regression-Align- Classification (CRAC) CRAC yields new state-of-the-art performances on many benchmarks. In experiments on seven benchmarks including OTB-2015, UAV123, NfS, VOT-2018, TrackingNet, GOT-10k and LaSOT, our CRACT exhibits very promising results in comparison with state-of-the-art competitors.
  arXiv  Detail & Related papers  (2020-11-25T02:18:33Z)
• A Differentiable Ranking Metric Using Relaxed Sorting Operation for Top-K Recommender Systems [1.2617078020344619]
  A recommender system generates personalized recommendations by computing the preference score of items, sorting the items according to the score, and filtering top-K items with high scores. While sorting and ranking items are integral for this recommendation procedure, it is nontrivial to incorporate them in the process of end-to-end model training. This incurs the inconsistency issue between existing learning objectives and ranking metrics of recommenders. We present DRM that mitigates the inconsistency and improves recommendation performance by employing the differentiable relaxation of ranking metrics.
  arXiv  Detail & Related papers  (2020-08-30T10:57:33Z)
• Controllable Multi-Interest Framework for Recommendation [64.30030600415654]
  We formalize the recommender system as a sequential recommendation problem. We propose a novel controllable multi-interest framework for the sequential recommendation, called ComiRec. Our framework has been successfully deployed on the offline Alibaba distributed cloud platform.
  arXiv  Detail & Related papers  (2020-05-19T10:18:43Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.