Do We Really Need to Drop Items with Missing Modalities in Multimodal Recommendation?

• URL: http://arxiv.org/abs/2408.11767v1
• Date: Wed, 21 Aug 2024 16:39:47 GMT
• Title: Do We Really Need to Drop Items with Missing Modalities in Multimodal Recommendation?
• Authors: Daniele Malitesta, Emanuele Rossi, Claudio Pomo, Tommaso Di Noia, Fragkiskos D. Malliaros,
• Abstract summary: We show that the lack of (some) modalities is, in fact, a widely-diffused phenomenon in multimodal recommendation. We propose a pipeline that imputes missing multimodal features in recommendation by leveraging traditional imputation strategies in machine learning.
• Score: 15.428850539237182
• License:
• Abstract: Generally, items with missing modalities are dropped in multimodal recommendation. However, with this work, we question this procedure, highlighting that it would further damage the pipeline of any multimodal recommender system. First, we show that the lack of (some) modalities is, in fact, a widely-diffused phenomenon in multimodal recommendation. Second, we propose a pipeline that imputes missing multimodal features in recommendation by leveraging traditional imputation strategies in machine learning. Then, given the graph structure of the recommendation data, we also propose three more effective imputation solutions that leverage the item-item co-purchase graph and the multimodal similarities of co-interacted items. Our method can be plugged into any multimodal RSs in the literature working as an untrained pre-processing phase, showing (through extensive experiments) that any data pre-filtering is not only unnecessary but also harmful to the performance.

Related papers

• Ducho meets Elliot: Large-scale Benchmarks for Multimodal Recommendation [9.506245109666907]
  Multi-faceted features characterizing products and services may influence each customer on online selling platforms differently. The common multimodal recommendation pipeline involves (i) extracting multimodal features, (ii) refining their high-level representations to suit the recommendation task, and (iv) predicting the user-item score. This paper settles as the first attempt to offer a large-scale benchmarking for multimodal recommender systems, with a specific focus on multimodal extractors.
  arXiv  Detail & Related papers  (2024-09-24T08:29:10Z)
• A Unified Graph Transformer for Overcoming Isolations in Multi-modal Recommendation [9.720586396359906]
  We argue that existing multi-modal recommender systems typically use isolated processes for both feature extraction and modality modelling. We propose a novel model, called Unified Multi-modal Graph Transformer (UGT), which leverages a multi-way transformer to extract aligned multi-modal features. We show that the UGT model can achieve significant effectiveness gains, especially when jointly optimised with the commonly-used multi-modal recommendation losses.
  arXiv  Detail & Related papers  (2024-07-29T11:04:31Z)
• Modality-Balanced Learning for Multimedia Recommendation [21.772064939915214]
  We propose a Counterfactual Knowledge Distillation method to solve the imbalance problem and make the best use of all modalities. We also design a novel generic-and-specific distillation loss to guide the multimodal student to learn wider-and-deeper knowledge from teachers. Our method could serve as a plug-and-play module for both late-fusion and early-fusion backbones.
  arXiv  Detail & Related papers  (2024-07-26T07:53:01Z)
• Multi-modal Crowd Counting via a Broker Modality [64.5356816448361]
  Multi-modal crowd counting involves estimating crowd density from both visual and thermal/depth images. We propose a novel approach by introducing an auxiliary broker modality and frame the task as a triple-modal learning problem. We devise a fusion-based method to generate this broker modality, leveraging a non-diffusion, lightweight counterpart of modern denoising diffusion-based fusion models.
  arXiv  Detail & Related papers  (2024-07-10T10:13:11Z)
• Attention-based sequential recommendation system using multimodal data [8.110978727364397]
  We propose an attention-based sequential recommendation method that employs multimodal data of items such as images, texts, and categories. The experimental results obtained from the Amazon datasets show that the proposed method outperforms those of conventional sequential recommendation systems.
  arXiv  Detail & Related papers  (2024-05-28T08:41:05Z)
• Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima [54.06000767038741]
  We analyze multimodal recommender systems from the novel perspective of flat local minima. We propose a concise yet effective gradient strategy called Mirror Gradient (MG) We find that the proposed MG can complement existing robust training methods and be easily extended to diverse advanced recommendation models.
  arXiv  Detail & Related papers  (2024-02-17T12:27:30Z)
• Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications [90.6849884683226]
  We study the challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds. We show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.
  arXiv  Detail & Related papers  (2023-06-07T15:44:53Z)
• Efficient Multimodal Fusion via Interactive Prompting [62.08292938484994]
  Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. We propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pre-trained transformers.
  arXiv  Detail & Related papers  (2023-04-13T07:31:51Z)
• Align and Attend: Multimodal Summarization with Dual Contrastive Losses [57.83012574678091]
  The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries. Existing methods fail to leverage the temporal correspondence between different modalities and ignore the intrinsic correlation between different samples. We introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input.
  arXiv  Detail & Related papers  (2023-03-13T17:01:42Z)
• Mining Latent Structures for Multimedia Recommendation [46.70109406399858]
  We propose a LATent sTructure mining method for multImodal reCommEndation, which we term LATTICE for brevity. We learn item-item structures for each modality and aggregates multiple modalities to obtain latent item graphs. Based on the learned latent graphs, we perform graph convolutions to explicitly inject high-order item affinities into item representations.
  arXiv  Detail & Related papers  (2021-04-19T03:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.