De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems

• URL: http://arxiv.org/abs/2501.05170v2
• Date: Tue, 22 Apr 2025 07:58:33 GMT
• Title: De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems
• Authors: Robin Burke, Gediminas Adomavicius, Toine Bogers, Tommaso Di Noia, Dominik Kowald, Julia Neidhardt, Özlem Özgöbek, Maria Soledad Pera, Nava Tintarev, Jürgen Ziegler,
• Abstract summary: We focus our discussion on the challenges of multistakeholder evaluation of recommender systems.<n>We discuss how to move from theoretical principles to practical implementation.<n>We aim to provide guidance to researchers and practitioners about incorporating these complex and domain-dependent issues of evaluation.
• Score: 10.731079374109596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, these systems cannot be evaluated strictly by the overall utility of a single stakeholder, as is often the case of more mainstream recommender system applications. In this article, we focus our discussion on the challenges of multistakeholder evaluation of recommender systems. We bring attention to the different aspects involved -- from the range of stakeholders involved (including but not limited to providers and consumers) to the values and specific goals of each relevant stakeholder. We discuss how to move from theoretical principles to practical implementation, providing specific use case examples. Finally, we outline open research directions for the RecSys community to explore. We aim to provide guidance to researchers and practitioners about incorporating these complex and domain-dependent issues of evaluation in the course of designing, developing, and researching applications with multistakeholder aspects.

Related papers

• Learning Recommender Systems with Soft Target: A Decoupled Perspective [49.83787742587449]
  We propose a novel decoupled soft label optimization framework to consider the objectives as two aspects by leveraging soft labels. We present a sensible soft-label generation algorithm that models a label propagation algorithm to explore users' latent interests in unobserved feedback via neighbors.
  arXiv  Detail & Related papers  (2024-10-09T04:20:15Z)
• Explainable Multi-Stakeholder Job Recommender Systems [0.0]
  New laws focus on aspects such as privacy, fairness, and explainability for recommender systems and AI at large. There is a need for a multi-stakeholder approach, as these systems are used by job seekers, recruiters, and companies simultaneously. I summarize my current research on the topic of explainable, multi-stakeholder job recommender systems and set out a number of future research directions.
  arXiv  Detail & Related papers  (2024-10-01T13:12:30Z)
• Review-based Recommender Systems: A Survey of Approaches, Challenges and Future Perspectives [11.835903510784735]
  Review-based recommender systems have emerged as a significant sub-field in this domain. We present a categorization of these systems and summarize the state-of-the-art methods, analyzing their unique features, effectiveness, and limitations. We propose potential directions for future research, including the integration of multimodal data, multi-criteria rating information, and ethical considerations.
  arXiv  Detail & Related papers  (2024-05-09T05:45:18Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors [68.68418801681965]
  We propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS.
  arXiv  Detail & Related papers  (2024-04-04T08:56:48Z)
• A Survey on Fairness-aware Recommender Systems [59.23208133653637]
  We present concepts of fairness in different recommendation scenarios, comprehensively categorize current advances, and introduce typical methods to promote fairness in different stages of recommender systems. Next, we delve into the significant influence that fairness-aware recommender systems exert on real-world industrial applications.
  arXiv  Detail & Related papers  (2023-06-01T07:08:22Z)
• Multi-Objective Recommender Systems: Survey and Challenges [5.076419064097734]
  We focus on developing machine learning algorithms that aim to predict which content is relevant for individual users. In real-world applications, however, optimizing the accuracy of such relevance predictions as a single objective is not sufficient. Multiple and often competing objectives have to be considered, leading to a need for more research in multi-objective recommender systems.
  arXiv  Detail & Related papers  (2022-10-19T05:51:54Z)
• Fairness in Recommender Systems: Research Landscape and Future Directions [119.67643184567623]
  We review the concepts and notions of fairness that were put forward in the area in the recent past. We present an overview of how research in this field is currently operationalized. Overall, our analysis of recent works points to certain research gaps.
  arXiv  Detail & Related papers  (2022-05-23T08:34:25Z)
• Experiments on Generalizability of User-Oriented Fairness in Recommender Systems [2.0932879442844476]
  A fairness-aware recommender system aims to treat different user groups similarly. We propose a user-centered fairness re-ranking framework applied on top of a base ranking model. We evaluate the final recommendations provided by the re-ranking framework from both user- (e.g., NDCG) and item-side (e.g., novelty, item-fairness) metrics.
  arXiv  Detail & Related papers  (2022-05-17T12:36:30Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)
• A Review on Pushing the Limits of Baseline Recommendation Systems with the integration of Opinion Mining & Information Retrieval Techniques [0.0]
  Recommendation Systems allow users to identify trending items among a community while being timely and relevant to the user's expectations. Deep Learning methods have been brought forward to achieve better quality recommendations. Researchers have tried to expand on the capabilities of standard recommendation systems to provide the most effective recommendations.
  arXiv  Detail & Related papers  (2022-05-03T22:13:33Z)
• Long-term Dynamics of Fairness Intervention in Connection Recommender Systems [5.048563042541915]
  We study a connection recommender system patterned after the systems employed by web-scale social networks. We find that, although seemingly fair in aggregate, common exposure and utility parity interventions fail to mitigate amplification of biases in the long term.
  arXiv  Detail & Related papers  (2022-03-30T16:27:48Z)
• Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation [87.82664566721917]
  This survey is based on more than 100 papers from top-tier conferences like IJCAI, AAAI, TheWebConf, Recsys, UMAP, and IUI.
  arXiv  Detail & Related papers  (2022-02-14T02:58:55Z)
• A Comprehensive Overview of Recommender System and Sentiment Analysis [1.370633147306388]
  This paper gives a comprehensive overview to help researchers who aim to work with recommender system and sentiment analysis. It includes a background of the recommender system concept, including phases, approaches, and performance metrics used in recommender systems. Then, it discusses the sentiment analysis concept and highlights the main points in the sentiment analysis, including level, approaches, and focuses on aspect-based sentiment analysis.
  arXiv  Detail & Related papers  (2021-09-18T01:08:41Z)
• Simultaneous Relevance and Diversity: A New Recommendation Inference Approach [81.44167398308979]
  We propose a new approach, which extends the general collaborative filtering (CF) by introducing a new way of CF inference, negative-to-positive. Our approach is applicable to a wide range of recommendation scenarios/use-cases at various sophistication levels. Our analysis and experiments on public datasets and real-world production data show that our approach outperforms existing methods on relevance and diversity simultaneously.
  arXiv  Detail & Related papers  (2020-09-27T22:20:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.