Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems

• URL: http://arxiv.org/abs/2305.15534v2
• Date: Fri, 26 May 2023 16:00:08 GMT
• Title: Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems
• Authors: Pedro Silva, Bhawna Juneja, Shloka Desai, Ashudeep Singh, Nadia Fawaz
• Abstract summary: We introduce end-to-end diversification to improve representation in search results and recommendations. We develop, experiment, and deploy scalable diversification mechanisms on the Pinterest platform. Our approaches significantly improve diversity metrics, with a neutral to a positive impact on utility metrics and improved user satisfaction.
• Score: 8.296711988456762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As the use of online platforms continues to grow across all demographics, users often express a desire to feel represented in the content. To improve representation in search results and recommendations, we introduce end-to-end diversification, ensuring that diverse content flows throughout the various stages of these systems, from retrieval to ranking. We develop, experiment, and deploy scalable diversification mechanisms in multiple production surfaces on the Pinterest platform, including Search, Related Products, and New User Homefeed, to improve the representation of different skin tones in beauty and fashion content. Diversification in production systems includes three components: identifying requests that will trigger diversification, ensuring diverse content is retrieved from the large content corpus during the retrieval stage, and finally, balancing the diversity-utility trade-off in a self-adjusting manner in the ranking stage. Our approaches, which evolved from using Strong-OR logical operator to bucketized retrieval at the retrieval stage and from greedy re-rankers to multi-objective optimization using determinantal point processes for the ranking stage, balances diversity and utility while enabling fast iterations and scalable expansion to diversification over multiple dimensions. Our experiments indicate that these approaches significantly improve diversity metrics, with a neutral to a positive impact on utility metrics and improved user satisfaction, both qualitatively and quantitatively, in production. An accessible PDF of this article is available at https://drive.google.com/file/d/1p5PkqC-sdtX19Y_IAjZCtiSxSEX1IP3q/view

Related papers

• A Preference-oriented Diversity Model Based on Mutual-information in Re-ranking for E-commerce Search [11.49911967350851]
  This paper proposes a Preference-oriented Diversity Model Based on Mutual-information (PODM-MI) PODM-MI consider both accuracy and diversity in the re-ranking process. We have successfully deployed PODM-MI on an e-commerce search platform.
  arXiv  Detail & Related papers  (2024-05-24T13:03:34Z)
• Diversify Question Generation with Retrieval-Augmented Style Transfer [68.00794669873196]
  We propose RAST, a framework for Retrieval-Augmented Style Transfer. The objective is to utilize the style of diverse templates for question generation. We develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward.
  arXiv  Detail & Related papers  (2023-10-23T02:27:31Z)
• Knowledge Graph Context-Enhanced Diversified Recommendation [53.3142545812349]
  This research explores the realm of diversified RecSys within the intricate context of knowledge graphs (KG) Our contributions include introducing an innovative metric, Entity Coverage, and Relation Coverage, which effectively quantifies diversity within the KG domain. In tandem with this, we introduce a novel technique named Conditional Alignment and Uniformity (CAU) which encodes KG item embeddings while preserving contextual integrity.
  arXiv  Detail & Related papers  (2023-10-20T03:18:57Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Graph Exploration Matters: Improving both individual-level and system-level diversity in WeChat Feed Recommender [21.0013026365164]
  Individual-level diversity and system-level diversity are both important for industrial recommender systems. We implement and deploy the combined system in WeChat App's Top Stories used by hundreds of millions of users.
  arXiv  Detail & Related papers  (2023-05-29T19:25:32Z)
• Performative Recommendation: Diversifying Content via Strategic Incentives [13.452510519858995]
  We show how learning can incentivize strategic content creators to create diverse content. Our approach relies on a novel form of regularization that anticipates strategic changes to content.
  arXiv  Detail & Related papers  (2023-02-08T21:02:28Z)
• Exploring Diversity in Back Translation for Low-Resource Machine Translation [85.03257601325183]
  Back translation is one of the most widely used methods for improving the performance of neural machine translation systems. Recent research has sought to enhance the effectiveness of this method by increasing the 'diversity' of the generated translations. This work puts forward a more nuanced framework for understanding diversity in training data, splitting it into lexical diversity and syntactic diversity.
  arXiv  Detail & Related papers  (2022-06-01T15:21:16Z)
• ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest [60.841761065439414]
  At Pinterest, we build a single set of product embeddings called ItemSage to provide relevant recommendations in all shopping use cases. This approach has led to significant improvements in engagement and conversion metrics, while reducing both infrastructure and maintenance cost.
  arXiv  Detail & Related papers  (2022-05-24T02:28:58Z)
• Sliding Spectrum Decomposition for Diversified Recommendation [6.448118871489599]
  We propose to study the diversity problem in such a scenario from an item sequence perspective using time series analysis techniques. We derive a method called sliding spectrum decomposition (SSD) that captures users' perception of diversity in browsing a long item sequence. We also share our experiences in designing and implementing a suitable item embedding method for accurate similarity measurement under long tail effect.
  arXiv  Detail & Related papers  (2021-07-12T05:41:54Z)
• DivAug: Plug-in Automated Data Augmentation with Explicit Diversity Maximization [41.82120128496555]
  Two factors regarding the diversity of augmented data are still missing: 1) the explicit definition (and thus measurement) of diversity and 2) the quantifiable relationship between diversity and its regularization effects. We propose a diversity measure called Variance Diversity and theoretically show that the regularization effect of data augmentation is promised by Variance Diversity. An unsupervised sampling-based framework, DivAug, is designed to directly maximize Variance Diversity and hence strengthen the regularization effect.
  arXiv  Detail & Related papers  (2021-03-26T16:00:01Z)
• On Compositions of Transformations in Contrastive Self-Supervised Learning [66.15514035861048]
  In this paper, we generalize contrastive learning to a wider set of transformations. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations.
  arXiv  Detail & Related papers  (2020-03-09T17:56:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.