A Multimodal Late Fusion Model for E-Commerce Product Classification

• URL: http://arxiv.org/abs/2008.06179v1
• Date: Fri, 14 Aug 2020 03:46:24 GMT
• Title: A Multimodal Late Fusion Model for E-Commerce Product Classification
• Authors: Ye Bi, Shuo Wang, Zhongrui Fan
• Abstract summary: In this study, we investigated a multimodal late fusion approach based on text and image modalities to categorize e-commerce products on Rakuten. Specifically, we developed modal specific state-of-the-art deep neural networks for each input modal, and then fused them at the decision level. Our team named pa_curis won the 1st place with a macro-F1 of 0.9144 on the final leaderboard.
• Score: 7.463657960984954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The cataloging of product listings is a fundamental problem for most e-commerce platforms. Despite promising results obtained by unimodal-based methods, it can be expected that their performance can be further boosted by the consideration of multimodal product information. In this study, we investigated a multimodal late fusion approach based on text and image modalities to categorize e-commerce products on Rakuten. Specifically, we developed modal specific state-of-the-art deep neural networks for each input modal, and then fused them at the decision level. Experimental results on Multimodal Product Classification Task of SIGIR 2020 E-Commerce Workshop Data Challenge demonstrate the superiority and effectiveness of our proposed method compared with unimodal and other multimodal methods. Our team named pa_curis won the 1st place with a macro-F1 of 0.9144 on the final leaderboard.

Related papers

• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• End-to-end multi-modal product matching in fashion e-commerce [0.6047429555885261]
  We present a robust multi-modal product matching system in an industry setting. We show how a human-in-the-loop process can be combined with model-based predictions to achieve near perfect precision.
  arXiv  Detail & Related papers  (2024-03-18T09:12:16Z)
• MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization [93.5217515566437]
  Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics. Existing MPS methods can produce promising results, but they still lack end-to-end product summarization. We propose an end-to-end multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce.
  arXiv  Detail & Related papers  (2023-08-22T11:00:09Z)
• MM-GEF: Multi-modal representation meet collaborative filtering [51.04679619309803]
  We propose a graph-based item structure enhancement method MM-GEF: Multi-Modal recommendation with Graph Early-Fusion. MM-GEF learns refined item representations by injecting structural information obtained from both multi-modal and collaborative signals.
  arXiv  Detail & Related papers  (2023-08-14T15:47:36Z)
• Multimodal E-Commerce Product Classification Using Hierarchical Fusion [0.0]
  The proposed method significantly outperformed the unimodal models' performance and the reported performance of similar models on our specific task. We did experiments with multiple fusing techniques and found, that the best performing technique to combine the individual embedding of the unimodal network is based on combining concatenation and averaging the feature vectors.
  arXiv  Detail & Related papers  (2022-07-07T14:04:42Z)
• Multi-Modal Attribute Extraction for E-Commerce [4.626261940793027]
  We develop a novel approach to seamlessly combine modalities, which is inspired by our single-modality investigations. Experiments on Rakuten-Ichiba data provide empirical evidence for the benefits of our approach.
  arXiv  Detail & Related papers  (2022-03-07T14:48:44Z)
• M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks [94.80043324367858]
  We contribute a large-scale dataset, named M5Product, which consists of over 6 million multimodal pairs. M5Product contains rich information of multiple modalities including image, text, table, video and audio.
  arXiv  Detail & Related papers  (2021-09-09T13:50:22Z)
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining [108.86502855439774]
  We investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval. We contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. We propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE)
  arXiv  Detail & Related papers  (2021-07-30T12:11:24Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.