End-to-end multi-modal product matching in fashion e-commerce

• URL: http://arxiv.org/abs/2403.11593v1
• Date: Mon, 18 Mar 2024 09:12:16 GMT
• Title: End-to-end multi-modal product matching in fashion e-commerce
• Authors: Sándor Tóth, Stephen Wilson, Alexia Tsoukara, Enric Moreu, Anton Masalovich, Lars Roemheld,
• Abstract summary: We present a robust multi-modal product matching system in an industry setting. We show how a human-in-the-loop process can be combined with model-based predictions to achieve near perfect precision.
• Score: 0.6047429555885261
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Product matching, the task of identifying different representations of the same product for better discoverability, curation, and pricing, is a key capability for online marketplace and e-commerce companies. We present a robust multi-modal product matching system in an industry setting, where large datasets, data distribution shifts and unseen domains pose challenges. We compare different approaches and conclude that a relatively straightforward projection of pretrained image and text encoders, trained through contrastive learning, yields state-of-the-art results, while balancing cost and performance. Our solution outperforms single modality matching systems and large pretrained models, such as CLIP. Furthermore we show how a human-in-the-loop process can be combined with model-based predictions to achieve near perfect precision in a production system.

Related papers

• Text-Based Product Matching -- Semi-Supervised Clustering Approach [9.748519919202986]
  This paper aims to present a new philosophy to product matching utilizing a semi-supervised clustering approach. We study the properties of this method by experimenting with the IDEC algorithm on the real-world dataset.
  arXiv  Detail & Related papers  (2024-02-01T18:52:26Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• UniMatch: A Unified User-Item Matching Framework for the Multi-purpose Merchant Marketing [27.459774494479227]
  We present a unified user-item matching framework to simultaneously conduct item recommendation and user targeting with just one model. Our framework results in significant performance gains in comparison with the state-of-the-art methods, with greatly reduced cost on computing resources and daily maintenance.
  arXiv  Detail & Related papers  (2023-07-19T13:49:35Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Cross-Modal Fine-Tuning: Align then Refine [83.37294254884446]
  ORCA is a cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. We show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities.
  arXiv  Detail & Related papers  (2023-02-11T16:32:28Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)
• Multimodal Adversarially Learned Inference with Factorized Discriminators [10.818838437018682]
  We propose a novel approach to generative modeling of multimodal data based on generative adversarial networks. To learn a coherent multimodal generative model, we show that it is necessary to align different encoder distributions with the joint decoder distribution simultaneously. By taking advantage of contrastive learning through factorizing the discriminator, we train our model on unimodal data.
  arXiv  Detail & Related papers  (2021-12-20T08:18:49Z)
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining [108.86502855439774]
  We investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval. We contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. We propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE)
  arXiv  Detail & Related papers  (2021-07-30T12:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.