Dataset and Models for Item Recommendation Using Multi-Modal User Interactions

• URL: http://arxiv.org/abs/2405.04246v1
• Date: Tue, 07 May 2024 12:03:22 GMT
• Title: Dataset and Models for Item Recommendation Using Multi-Modal User Interactions
• Authors: Simone Borg Bruun, Krisztian Balog, Maria Maistro,
• Abstract summary: We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels. In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. We propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space.
• Score: 14.054250597878465
• License:
• Abstract: While recommender systems with multi-modal item representations (image, audio, and text), have been widely explored, learning recommendations from multi-modal user interactions (e.g., clicks and speech) remains an open problem. We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels (website and call center). In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. To address these challenges, we publish a real-world dataset that allows progress in this under-researched area. We further present and benchmark various methods for leveraging multi-modal user interactions for item recommendations, and propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space. Our analysis reveals important interactions between the different modalities and that a frequently occurring modality can enhance learning from a less frequent one.

Related papers

• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout. DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations [17.409790984399052]
  This paper explores the impact of multi-modal interactions, which incorporate images and audio alongside text, on user engagement. Our findings reveal a significant enhancement in user engagement with multi-modal interactions compared to text-only dialogues. Results suggest that multi-modal interactions optimize cognitive processing and facilitate richer information comprehension.
  arXiv  Detail & Related papers  (2024-06-21T09:26:55Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications [90.6849884683226]
  We study the challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds. We show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.
  arXiv  Detail & Related papers  (2023-06-07T15:44:53Z)
• Multi-Behavior Sequential Recommendation with Temporal Graph Transformer [66.10169268762014]
  We tackle the dynamic user-item relation learning with the awareness of multi-behavior interactive patterns. We propose a new Temporal Graph Transformer (TGT) recommendation framework to jointly capture dynamic short-term and long-range user-item interactive patterns.
  arXiv  Detail & Related papers  (2022-06-06T15:42:54Z)
• Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network [25.563806871858073]
  This work proposes a Memory-Augmented Transformer Networks (MATN) to enable the recommendation with multiplex behavioral relational information. In our MATN framework, we first develop a transformer-based multi-behavior relation encoder, to make the learned interaction representations be reflective of the cross-type behavior relations. A memory attention network is proposed to supercharge MATN capturing the contextual signals of different types of behavior into the category-specific latent embedding space.
  arXiv  Detail & Related papers  (2021-10-08T09:54:43Z)
• Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendation [56.12499090935242]
  This work proposes a Knowledge-Enhanced Hierarchical Graph Transformer Network (KHGT) to investigate multi-typed interactive patterns between users and items in recommender systems. KHGT is built upon a graph-structured neural architecture to capture type-specific behavior characteristics. We show that KHGT consistently outperforms many state-of-the-art recommendation methods across various evaluation settings.
  arXiv  Detail & Related papers  (2021-10-08T09:44:00Z)
• Graph Meta Network for Multi-Behavior Recommendation [24.251784947151755]
  We propose a Multi-Behavior recommendation framework with Graph Meta Network to incorporate the multi-behavior pattern modeling into a meta-learning paradigm. Our developed MB-GMN empowers the user-item interaction learning with the capability of uncovering type-dependent behavior representations.
  arXiv  Detail & Related papers  (2021-10-08T08:38:27Z)
• Modeling High-order Interactions across Multi-interests for Micro-video Reommendation [65.16624625748068]
  We propose a Self-over-Co Attention module to enhance user's interest representation. In particular, we first use co-attention to model correlation patterns across different levels and then use self-attention to model correlation patterns within a specific level.
  arXiv  Detail & Related papers  (2021-04-01T07:20:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.