Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?

• URL: http://arxiv.org/abs/2307.09985v3
• Date: Sun, 24 Mar 2024 15:53:57 GMT
• Title: Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?
• Authors: Yu-chen Fan, Yitong Ji, Jie Zhang, Aixin Sun,
• Abstract summary: We conduct a meticulous analysis of the MovieLens dataset. There are significant differences in user interactions at the different stages when a user interacts with the MovieLens platform. We discuss the discrepancy between the interaction generation mechanism that is employed by the MovieLens system and that of typical real-world recommendation scenarios.
• Score: 43.3971105361606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A typical benchmark dataset for recommender system (RecSys) evaluation consists of user-item interactions generated on a platform within a time period. The interaction generation mechanism partially explains why a user interacts with (e.g., like, purchase, rate) an item, and the context of when a particular interaction happened. In this study, we conduct a meticulous analysis of the MovieLens dataset and explain the potential impact of using the dataset for evaluating recommendation algorithms. We make a few main findings from our analysis. First, there are significant differences in user interactions at the different stages when a user interacts with the MovieLens platform. The early interactions largely define the user portrait which affects the subsequent interactions. Second, user interactions are highly affected by the candidate movies that are recommended by the platform's internal recommendation algorithm(s). Third, changing the order of user interactions makes it more difficult for sequential algorithms to capture the progressive interaction process. We further discuss the discrepancy between the interaction generation mechanism that is employed by the MovieLens system and that of typical real-world recommendation scenarios. In summary, the MovieLens platform demonstrates an efficient and effective way of collecting user preferences to address cold-starts. However, models that achieve excellent recommendation accuracy on the MovieLens dataset may not demonstrate superior performance in practice, for at least two kinds of differences: (i) the differences in the contexts of user-item interaction generation, and (ii) the differences in user knowledge about the item collections. While results on MovieLens can be useful as a reference, they should not be solely relied upon as the primary justification for the effectiveness of a recommendation system model.

Related papers

• EDGE-Rec: Efficient and Data-Guided Edge Diffusion For Recommender Systems Graphs [0.0]
  We propose a new attention mechanism to take advantage of real-valued interaction weights as well as user and item features directly. We train a novel Graph Diffusion Transformer GDiT architecture to iteratively denoise the weighted interaction matrix of the user-item interaction graph directly. Inspired by the recent progress in text-conditioned image generation, our method directly produces user-item rating predictions on the same scale as the original ratings.
  arXiv  Detail & Related papers  (2024-09-23T03:23:20Z)
• Topology-aware Debiased Self-supervised Graph Learning for Recommendation [6.893289671937124]
  We propose Topology-aware De Self-supervised Graph Learning ( TDSGL) for recommendation. TDSGL constructs contrastive pairs according to the semantic similarity between users (items) Our results show that the proposed model outperforms the state-of-the-art models significantly on three public datasets.
  arXiv  Detail & Related papers  (2023-10-24T14:16:19Z)
• COLA: Improving Conversational Recommender Systems by Collaborative Augmentation [9.99763097964222]
  We propose a collaborative augmentation (COLA) method to improve both item representation learning and user preference modeling. We construct an interactive user-item graph from all conversations, which augments item representations with user-aware information. To improve user preference modeling, we retrieve similar conversations from the training corpus, where the involved items and attributes that reflect the user's potential interests are used to augment the user representation.
  arXiv  Detail & Related papers  (2022-12-15T12:37:28Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)
• Multi-Behavior Sequential Recommendation with Temporal Graph Transformer [66.10169268762014]
  We tackle the dynamic user-item relation learning with the awareness of multi-behavior interactive patterns. We propose a new Temporal Graph Transformer (TGT) recommendation framework to jointly capture dynamic short-term and long-range user-item interactive patterns.
  arXiv  Detail & Related papers  (2022-06-06T15:42:54Z)
• Relation-aware Heterogeneous Graph for User Profiling [24.076585294260816]
  We propose to leverage the relation-aware heterogeneous graph method for user profiling. We adopt the query, key, and value mechanism in a transformer fashion for heterogeneous message passing. We conduct experiments on two real-world e-commerce datasets and observe a significant performance boost of our approach.
  arXiv  Detail & Related papers  (2021-10-14T06:59:30Z)
• Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendation [56.12499090935242]
  This work proposes a Knowledge-Enhanced Hierarchical Graph Transformer Network (KHGT) to investigate multi-typed interactive patterns between users and items in recommender systems. KHGT is built upon a graph-structured neural architecture to capture type-specific behavior characteristics. We show that KHGT consistently outperforms many state-of-the-art recommendation methods across various evaluation settings.
  arXiv  Detail & Related papers  (2021-10-08T09:44:00Z)
• From Implicit to Explicit feedback: A deep neural network for modeling sequential behaviours and long-short term preferences of online users [3.464871689508835]
  Implicit and explicit feedback have different roles for a useful recommendation. We go from the hypothesis that a user's preference at a time is a combination of long-term and short-term interests.
  arXiv  Detail & Related papers  (2021-07-26T16:59:20Z)
• Dynamic Graph Collaborative Filtering [64.87765663208927]
  Dynamic recommendation is essential for recommender systems to provide real-time predictions based on sequential data. Here we propose Dynamic Graph Collaborative Filtering (DGCF), a novel framework leveraging dynamic graphs to capture collaborative and sequential relations. Our approach achieves higher performance when the dataset contains less action repetition, indicating the effectiveness of integrating dynamic collaborative information.
  arXiv  Detail & Related papers  (2021-01-08T04:16:24Z)
• Optimizing Interactive Systems via Data-Driven Objectives [70.3578528542663]
  We propose an approach that infers the objective directly from observed user interactions. These inferences can be made regardless of prior knowledge and across different types of user behavior. We introduce Interactive System (ISO), a novel algorithm that uses these inferred objectives for optimization.
  arXiv  Detail & Related papers  (2020-06-19T20:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.