Scenario-Wise Rec: A Multi-Scenario Recommendation Benchmark

• URL: http://arxiv.org/abs/2412.17374v2
• Date: Thu, 16 Jan 2025 04:40:18 GMT
• Title: Scenario-Wise Rec: A Multi-Scenario Recommendation Benchmark
• Authors: Xiaopeng Li, Jingtong Gao, Pengyue Jia, Xiangyu Zhao, Yichao Wang, Wanyu Wang, Yejing Wang, Yuhao Wang, Xiangyu Zhao, Huifeng Guo, Ruiming Tang,
• Abstract summary: We introduce our benchmark, textbfScenario-Wise Rec, which comprises 6 public datasets and 12 benchmark models, along with a training and evaluation pipeline. We aim for this benchmark to offer researchers valuable insights from prior work, enabling the development of novel models.
• Score: 54.93461228053298
• License:
• Abstract: Multi Scenario Recommendation (MSR) tasks, referring to building a unified model to enhance performance across all recommendation scenarios, have recently gained much attention. However, current research in MSR faces two significant challenges that hinder the field's development: the absence of uniform procedures for multi-scenario dataset processing, thus hindering fair comparisons, and most models being closed-sourced, which complicates comparisons with current SOTA models. Consequently, we introduce our benchmark, \textbf{Scenario-Wise Rec}, which comprises 6 public datasets and 12 benchmark models, along with a training and evaluation pipeline. Additionally, we validated the benchmark using an industrial advertising dataset, reinforcing its reliability and applicability in real-world scenarios. We aim for this benchmark to offer researchers valuable insights from prior work, enabling the development of novel models based on our benchmark and thereby fostering a collaborative research ecosystem in MSR. Our source code is also publicly available.

Related papers

• Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines [64.61315565501681]
  Multi-modal Retrieval Augmented Multi-modal Generation (M$2$RAG) is a novel task that enables foundation models to process multi-modal web content. Despite its potential impact, M$2$RAG remains understudied, lacking comprehensive analysis and high-quality data resources.
  arXiv  Detail & Related papers  (2024-11-25T13:20:19Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline [78.00363373925758]
  We study the features of the BPR model, indicating their impact on its performance, and investigate open-source BPR implementations. Our analysis reveals inconsistencies between these implementations and the original BPR paper, leading to a significant decrease in performance of up to 50% for specific implementations. We show that the BPR model can achieve performance levels close to state-of-the-art methods on the top-n recommendation tasks and even outperform them on specific datasets.
  arXiv  Detail & Related papers  (2024-09-21T18:39:53Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [69.4501863547618]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios. With a focus on factual accuracy, we propose three novel metrics Completeness, Hallucination, and Irrelevance. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• Adaptive Utilization of Cross-scenario Information for Multi-scenario Recommendation [11.489766641148151]
  Multi-scenario Recommendation (MSR) is an important topic that improves ranking performance by leveraging information from different scenarios. Recent methods for MSR mostly construct scenario shared or specific modules to model commonalities and differences among scenarios. We propose a unified model named Cross-Scenario Information Interaction (CSII) to serve all scenarios by a mixture of scenario-dominated experts.
  arXiv  Detail & Related papers  (2024-07-29T06:17:33Z)
• A Unified Search and Recommendation Framework Based on Multi-Scenario Learning for Ranking in E-commerce [13.991015845541257]
  We propose an effective and universal framework for Unified Search and Recommendation (USR) USR can be applied to various multi-scenario models and significantly improve their performance. USR has been successfully deployed in the 7Fresh App.
  arXiv  Detail & Related papers  (2024-05-17T14:57:52Z)
• Multi-document Summarization: A Comparative Evaluation [0.0]
  This paper is aimed at evaluating state-of-the-art models for Multi-document Summarization (MDS) on different types of datasets in various domains. We analyzed the performance of PRIMERA and PEG models on Big-Survey and MS$2$ datasets.
  arXiv  Detail & Related papers  (2023-09-10T07:43:42Z)
• UMSE: Unified Multi-scenario Summarization Evaluation [52.60867881867428]
  Summarization quality evaluation is a non-trivial task in text summarization. We propose Unified Multi-scenario Summarization Evaluation Model (UMSE) Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios.
  arXiv  Detail & Related papers  (2023-05-26T12:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.